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ABSTRACT 



This thesis evaluates the Navy's test and evaluation (T&E) process 
for air-launched guided missiles, identifies T&E management problems, 
and proposes utilization of a particular test program strategy. Many 
changes have been made to improve the T&E process, but some of these 
have resulted in costly inefficiencies. Contributing problems include: 
the operational test agencies do not participate adequately in early 
test planning, excess duplication exists in testing done by major par- 
ticipants in the Navy air-launched guided missile T&E process, lack of 
definition of a mission profile leads to improper and inadequate test- 
ing of air-launched guided missiles, and numerous other T&E management 
problems . 

The utilization of an Integrated Test Program approach for air- 
launched guided missile T&E is proposed. Test data from contractor 
demonstration tests, limited TECHEVAL, and OPEVAL would be pooled to 
confirm compliance with specification requirements and verify opera- 
tional effectiveness and suitability. Test assets and other resources 
would be conserved and overall T&E schedule improved. 
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I . INTRODUCTION 



A. BACKGROUND 

The acquisition process for defense weapon systems came under in- 
tense criticism in the late 1960s and early 1970s because of costly 
development overruns, schedule slippage, and system performance short- 
comings. The President's Blue Ribbon Defense Panel, the General 
Accounting Office, the Congress's Commission on Government Procurement 
and the Armed Services Committees of the House of Representatives and 
Senate have consistently identified test and evaluation as a major 
problem area. The Department of Defense (DoD) responded to the criti- 
cism by issuing new guidelines and directives instituting procedures 
which put a new perspective on the acquisition process. A major out- 
come of the new procedures was the increased emphasis of test and 
evaluation (T&E) . Of concern was the fact that T&E had not commanded 
the importance, stature, or priority that it must have if it is to be 
a primary source of information on the progress of major defense sys- 
tems and for decision-making. 

As a result, the new directives stated that T&E will commence as 
early as possible and be conducted throughout the system acquisition 
process, and that program schedules and milestone decisions will be 
based upon accomplishment and assessment of the program's T&E efforts. 
Also implemented was the establishment of a T&E organization in the 
Offices of the Secretary of Defense (OSD) to oversee defense T&E activ 
ities. It was further required that a primary role would be played by 
independent test agencies within DoD components in the accomplishment 
of T&E [Refs. 1 and 2]. 
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Implementation of the guidelines and directives has encountered 
some difficulties. TSE is costly, difficult, and time consuming. 

There are many people involved and genuine competition exists within 
DoD between agencies, between DoD agencies and contractors, and be- 
tween contractors for scarce defense dollars. 

However, the value of TSE cannot be overlooked. In a recent over- 
view statement to Congress, the Deputy Director (Test and Evaluation) 
of the Office of Defense Research and Engineering said that TSE was a 
good investment from which essential information was obtained to 
effectively manage the research, development, and procurement of 
defense systems. By insuring that only proven, fully-ef fective weapon 
systems are deployed, TSE allows us to obtain maximum military capa- 
bility for our defense dollars [Ref. 3] . 

B. PURPOSE A1JD OBJECTIVES 

The purpose of this thesis is to examine the management policy of 
TSE during this new era of TSE emphasis and some of its associated 
problem areas, and then to relate the policy and problems to an Air 
Launched Guided Missile (ALGM) program. 

The primary objective of the thesis is to provide guidance for an 
optimizing test strategy for TSE of ALGM programs to be used by the 
program manager in his management duties. In addition to providing a 
test strategy, the thesis is intended to be a self-contained document 
wherein the ALGM program manager has the TSE policy, requirements, in- 
structions, organizations, facilities, potential problem areas, test- 
ing requirements and typical test program scope identified. This 
should help the program manager resolve some of the TSE management 
issues and help him avoid some of the problems inherent in the TSE 
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efforts. The test strategy will aid the program manager in the budget- 
ing process , in his early program planning, in bringing T&E issues 
into focus, and will serve as a source material in preparation of such 
program documents as Decision Coordinating Papers and Test and Evalua- 
tion Master Plans. 

C. SCOPE 

The thesis addresses T&E from general policy within DoD to Navy 
application and specific Navy ALG11 program implementation. By the 
nature of the subject matter (management of T&E) , many of the manage- 
ment issues and problems are broadly applicable and relate to the total 
spectrum of T&E. However, the subject can be applied equally to ALGM 
defense weapon systems. 

The thesis investigates management of T&E in reference to Navy 
Acquisition Category I and II ALGM programs. 1 The program life cycle 
period addressed is from program initiation through release to produc- 
tion (that period covered by Research, Development, Test and Evaluation 
(RDT&E) funding) . 

An Air-Launched Guided Missile system may be a constituent of a 
larger defense weapon system which has as its primary sub-system a 
missile, aircraft avionics sub-system, and support equipment. The 
missile is dominant in the system and is in the center of most T&E 
activity. The missile is constructed of sections, these typically be- 
ing guidance, control, warhead and propulsion. Within each missile 



x The Navy Acquisition Categories (ACAT) are defined in reference 4. 
ACAT I is for systems with dollar thresholds of $75 M RDT&E and $300 M 
production, and ACAT II have $20 M RDT&E and $50 M production 
thresholds . 
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section are components , assemblies, and parts. ALGMs currently within 
the Navy inventory are comprised of two broad categories or groups, 
air-to-air and air-to-surface . Typical within the air-to-air group 
are Sidewinder, Sparrow, and Phoenix. The air-to-surface group includes 
Condor, Shrike, Standard Arm, and HARM. 

D. ISSUES 

The major issue addressed in this thesis is how can TSE be managed 
effectively and efficiently for ALGM programs. Related issues include 
whether there is an optimal test strategy which minimizes proliferation 
of requirements , resources, and facilities, yet provides resolution to 
development test questions and provides significant and relevant infor- 
mation to support milestone decisions. Also, can the problems inherent 
to TSE be reduced by implementation and procedural changes or adjust- 
ments? Specific areas of interest include: what TSE is required, de- 

sired, and affordable; who performs the TSE and when is it accomplished; 
what is the best approach to implement to satisfy the ALGM TSE require- 
ments, yet is within program dollar and schedule constraints? 

These and similar issues relating to the management of an ALGM pro- 
gram's TSE activities are continually addressed in the thesis and 
potential solutions are explored. 

E. RESEARCH METHOD 

A data and literature search was performed initially, and all perti- 
nent DoD directives, Navy instructions, and other TSE guidelines reviewed. 
TSE facilities and organizations which might have a role in an ALGM 
program were examined. 

Interviews were conducted with participants in ALGM TSE efforts 
(Appendix A) , both those in upper level management and review cycles 
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and those who directly implement program T&E activities. Activities 
interviewed include: 

1. Deputy Director of Defense Research and Engineering, Test and 
Evaluation (DD (T&E) ) 

2. Offices of the Chief of Naval Operations (OPNAV) 

3. Naval Material Command (NAVMAT) 

4. Naval Air Systems Command (NAVAIR) 

5. Operational Test and Evaluation Force (OPTEVFOR) 

6. Navy Laboratories 

7. Participating Field Activities 

8. Development Contractors 
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II. TEST AND EVALUATION AND THE SYSTEM LIFE CYCLE 



A* NATURE AND PURPOSE OF T&E 

The terms "test" and "evaluation” symbolize distinguishable func- 
tions in the RDT&E process. "Test” denotes the actual testing of hard- 
ware/software (models, prototypes, production equipment, computer pro- 
grams) to obtain data valuable in developing new capabilities , managing 
the developing activities, or making decisions at program milestones 
or on the allocation of resources. "Evaluation” denotes the process 
whereby the information content in the data is logically assembled and 
analyzed to aid in making systematic decisions. In a broad sense, TSE 
may be defined as the physical testing, experimentation and analyses 
performed during the course of research, development, introduction and 
employment of a weapon system or sub-system, and the analytical or 
evaluative studies performed using the data generated [Ref. 5] . 

T&E is an integral part of all phases of the development of systems 
and equipments and provides information for a number of purposes and 
several different classes of information users. T&E provides the 
following to the users and developers of systems; 

1. Information for development 

2. Information for acquisition milestone decisions 

3. Information for effective operational utilization 

4. Information for assessment of acquisition risks 

These are discussed briefly. 

1 . Information for Development 

Testing of systems under development is an inherent part of the 
Research and Development (R&D) process through which technical 
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deficiencies and uncertainties , as well as design and operation prob- 
lems are identified and resolved. This testing provides design data 
feedback to the developing agency and development contractors. The 
data feedback is also useful in the design- test-evaluate-redesign pro- 
cess which is basic to the development of reliable systems [Ref. 5] . 

2 . Information for Acquisition Milestone Decisions 

The major milestone decisions in the development of a system, 
to initiate advanced development, conduct full-scale development, and 
produce a system, are by nature investment decisions. The issue at 
these milestone decision points is whether initiating or continuing the 
acquisition will result in the most productive use of the resources 
(money, material, and personnel) . T&E helps provide the basis for 
these decisions, and the best information possible must be obtained to 
aid the decision process. In DoD Directive 5000.3 the statement is 
made that the basis for decisions to commit added resources to a pro- 
gram shall be successful accomplishment of the T&E objectives [Refs. 

5 and 6] . 

The Deputy Director (Test and Evaluation) , Office of the 
Director of Defense Research and Engineering, in his overview statement 
to Congress in March 1977, stated that the primary purpose of T&E is to 
provide information for program-level decision making. Two important 
questions addressed, using T&E information as a basis, are "How well 
current objectives are being met and what is likely to be the ultimate 
outcome of the program." Note, one question deals with the past and 
the other with the future [Ref. 3] . 

3 . Information for Effective Operational Utilization 

Information obtained both through development and operational 

testing provides a valuable data base to the operational user. Typical 
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is data obtained by the exploration of the performance envelopes of 
the system. With this information, tactics can be developed and 
doctrine established for the most effective utilization of the system. 
4. Information for Assessment of Acquisition Risks 

Unfortunately, many times T&E does not provide information 
which leads to clear-cut decisions. The T&E information does, however, 
provide the decision-maker with a means to make judgments and to assess 
the associated program risks, whether they be technical, managerial, 
cost or schedule. DoDD 5000.3 states directly that T&E shall be con- 
ducted as necessary to provide the information to assess the programs 
acquisition risks [Ref. 6] . 

B. DOD POLICY FOR T&E 

The current DoD policy for T&E has evolved from the changes and 
direction initiated in 1969 by David Packard, Deputy Secretary of 
Defense, when he began to institute reforms in the system acquisition 
process. Among these reforms was that needs and proposed solutions 
should be validated, preferably by experimental verification (testing) . 
Mr. Packard proposed early demonstration in areas possessing high 
development risk, emphasizing T&E as being of paramount importance 
[Ref. 7]. A study of the major weapon system acquisition process, 
also in 1969, by a Blue Ribbon Defense Panel established by the Presi- 
dent and Secretary of Defense included the subject of T&E [Ref. 8] . 

This study concentrated, however, on OT&E and essentially found it to 

be unsatisfactory, recommended changes to make it more effective, and 

... 2 

concurred with its significance in the system acquisition process. 



2 

For a more detailed account of this study, as well as other studies 
on T&E and the role of OT&E in the system acquisition process, see 
thesis by W. C. Bowes [Ref. 9] . 
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From this study and the implementation of Mr. Packard’s philosophy, 

DoD Directive 5000.1 was issued in 1971 containing requirements for 
acquisition of major defense systems. Addressing T&E policy, 

DoDD 5000.1 stated: 

"Test and evaluation shall commence as early as possible. A 
determination of operational suitability, including logistic support 
requirements, will be made prior to large-scale production commit- 
ments, making use of the most realistic test environment possible 
and the best representation of the future operational system avail- 
able. The results of this operational testing will be evaluated 
and presented to the DSARC at the time of the production decision" 
[Ref. 1]. 

DoD then responded with DoD Directive 5000.3 in 1973 [Ref. 2] which 
contained the requirements for test and evaluation. 

A similar statement regarding T&E policy is contained in the 
January 1977 revision of DoDD 5000.1 [Ref. 10]. Thus this directive 
provides the basic foundation for T&E as it is implemented today. 

DoD Directive 5000.3 establishes the policy for the conduct of test 
and evaluation by the Military Departments and Defense Agencies (DoD 
Components) in the acquisition of defense systems. This document con- 
tains general policies and principles for T&E. It states that T&E 
shall be commenced as early as possible and conducted throughout the 
system acquisition process, and that acquisition schedules will be 
based upon accomplishing T&E milestones prior to decision-making. 

Thus the T&E program is planned and executed concurrently with the 
development phases to support the various users of the T&E information. 
DoDD 5000.3 covers the requirements for T&E, its function and its role. 
It also specifies the requirement for the generation of an overall T&E 
plan and contains procedures and requirements for T&E inputs to the 
major program review with the Defense System Acquisition Review Council 
(DSARC) and into the program's Decision Coordinating Paper (DCP) . Of 
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importance to note is that the initial DCP presented to the DSARC at 
Milestone I will identify the critical questions and areas of risk to 
be resolved by T&E , and provide test objectives, schedules and mile- 
stones, In addition, revised DCPs at later milestones will give re- 
sults of T&E accomplished to date and update all other T&E areas. The 
DSARC at major milestones is charged with the responsibility to assess 
and comment to SECDEF as to the adequacy of the test results to support 
a decision to proceed with development. The DSARC input on T&E is 
obtained from the Office of the Deputy Director (Test and Evaluation) 
whose function is to oversee defense T&E activities and set policy for 
conduct of T&E [Ref, 6] . 

DoDD 5000,3 also contains policy for T&E planning in DoD. It states 

"The DoD Component will prepare as early as possible in the 
acquisition process, and prior to initiation of Full-Scale Develop- 
ment an overall test and evaluation plan to identify and integrate 
the effort and schedules of all T&E to be accomplished and to 
insure that all necessary T&E is accomplished prior to the key 
decision points. The TEMP (T&E Master Plan) will be kept current 
by the DoD Component" [Ref, 6], 

A significant feature of the T&E policy promulgated by DoDD 5000,1 
and 5000.3 is the increased importance and strength of the independent 
test agency. This evolved in response to the previously cited Blue 
Ribbon Defense Panel study and was concurred with by the study conducted 
by the Commission on Government Procurement [Refs. 8 and 11], 

C. RELATIONSHIP TO SYSTEM LIFE CYCLE 

Figure 1 depicts the system life cycle as adapted from Ref. 7. The 
process begins with requirements or needs and evolves through the 
development phases to production and operational use. Figure 2 shows 
the T&E areas corresponding to the life cycle phase for defense acquisi- 
tion systems according to T&E types. Also shown are the major program 
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Figure 1. System Life Cycle (Adapted From Ref. 7) 



23 



Program Program Full-Scale First Major 




24 



Figure 2. Weapon System Life Cycle and T&E Phases (Ref. 4) 



milestone decision points. T&E plays a vital part in Milestones I , II 
and III. Milestone 0 is the program initiation point where a mission 
need has been identified and exploration of alternative solutions 
authorized. Milestone I is the program validation decision point where 
selected alternatives continue into a demonstration and validation 
effort. A DCP is required at this tine, containing, among other things, 

identified T&E objectives and issues. Milestone II is the point where 

a commitment to full-scale development is made for the preferred system. 
Demonstration and validation of T&E objectives should have been com- 
pleted and the results should support this decision. The DCP is updated 

appropriately. Milestone III is the point where the decision is made to 

produce and deploy the system. T&E makes a major input at this program 
milestone point [Refs. 4, 10 and 12] . 

There are three types of T&E used in the system life cycle: 

1. Development Test and Evaluation (DT&E ) 

2. Operational Test and Evaluation (OT&E) which is composed of: 

a. Initial OT&E (IOT&E) 

b. Follow-on OT&E (FOT&E ) 

3. Production Acceptance Test and Evaluation (PAT&E) 

References 4 and 6 describe in detail the three types of T&E. 

Appendix B contains a brief description of each type for quick reference. 

D. PROGRAM REVIEWS 

As noted in section II B, a principal purpose of T&E is to provide 
assistance to decision-makers at program reviews or milestones by pro- 
viding data and information. Other important program reviews, keyed 
especially to the commitment of resources, rely much on T&E information. 
Examples are Congressional reviews with the House and Senate Armed 
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Services and Appropriations Committees. At these reviews, T&E results, 
in addition to other program inputs, are used to determine the commit- 
ment of funds, program continuation or program alteration. For major 
programs, reviews occur at various upper management levels throughout 
the organization, and at which T&E results are vital. 

In addition to upper level program reviews, T&E plays an important 
role in reviews held throughout the system life cycle at the develop- 
ment organization level. Design reviews occur as necessary during the 
acquisition process, involving the development agency, the development 
contractor, and the applicable T&E agencies and activities. Testing 
data and results are an inherent and important part of these reviews, 
which provide valuable design and development information. Also occur- 
ring at the development organization level are periodic reviews of the 
T&E program to examine test progress and for review of problem identi- 
fication and resolution information. 
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III. NAVY TEST AND EVALUATION REQUIREMENTS 
AND PROGRAM STRUCTURE 



This section contains the Navy's implementation of the DoD direc- 
tives for T&E and, where appropriate, presents unique features relative 
to ALGM programs . 

A. NAVY POLICY FOR T&E 

The Navy's most recent implementation response to DoDD 5000.3 was 
OPNAVINST 3960.10, issued in October 1975, which established the policy 
for T&E in Navy acquisition programs. This is the principal Navy policy 
document for T &E. OPNAVINST 3960.10 defines the T&E responsibilities 
of Navy activities and indicates procedures for planning, conducting, 
and reporting T &S. It delineates the complementary relationship between 
DT&E and OT&E, and establishes procedures and format requirements for 
Test and Evaluation Master Plans (TEMPs) [Ref. 4] . The TEMP is the 
controlling management document which defines T&E for each acquisition 
program. OPNAVINST 3960.10 must be understood thoroughly by the program 
manager, his designee for T&E , and others responsible for implementing 
T&e policy on a program. 

OPNAVINST 3960.10 also clarifies the responsibility of the indepen- 
dent test agency. The agency for the Navy, OPTEVFOR, and its commander, 
COMOPTEVFOR , has been assigned this responsibility. 

The Department of Navy RDT&E Management Guide [Ref. 5] also provides 
a valuable guideline on management of Navy T&E . Chapter VII and 
Appendix G of Ref. 5 are devoted totally to Navy T&E and provide 
appropriate amplification. 
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3. MANAGEMENT PHILOSOPHY AND ORGANIZATION 



The requirement for management of a Navy program's T&E efforts 
rests primarily with the Program Manager. Even though a broad organi- 
zation exists in Navy T&E , the Program Manager is responsible for 
planning , implementing and reporting the T&E for the weapon system. 
OPNAVTNST 3960.10 points out that the Program Manager within the 
Developing Agency is responsible for development and execution of an 
adequate T&E program. His T&E responsibilities include defining a 
test program which illuminates test issues and problems; preparation 
and updating of the TEMP; and arranging for performance of the required 
and planned testing and subsequent evaluation [Ref. 4] . 

In the conduct of the T&E program, however, OPNAVINST 3960.10 
delegates authority for the three types of testing to two different 
organizations. The Developing Agency is responsible for DT&E. For 
ALGMs, the Developing Agency is also responsible for PAT&E . This in- 
cludes planning, conducting and reporting. In addition the Developing 
Agency maintains liaison with COMOPTEVFOR concerning the DT&E program 
and provides COMOPTEVFOR with appropriate results. For OT&E, the Navy 
has assigned the responsibility to OPTEVFOR, which is an organization 
separate and distinct from the developing and procuring command, and 
from the using command (although attached to CNO) . OT&E is planned and 
conducted by OPTEVFOR, and results are reported directly to the CNO 
by COMOPTEVFOR. In addition, OPTEVFOR provides the Developing Agency 
with all significant OT&E results [Ref. 4] . 

An organizational chart which depicts the Navy's T&E agencies 
(along with that at the 3ECDEF level) is shown in Figure 3. It also 
shows those field activities involved in ALGM program. Descriptions 
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Figure 3. Present Navy T&E Organization Structure for ALGM Systems 




of each organization's functions and responsibilities can be found in 
Ref, 5. 

Although complementary, DT&E and OT&E are indeed separate and dis- 
tinct, both in their basic goals and objectives and in their implemen- 
tation and conduct. From their requirements foundation, DT&E works to 
specifications stated in functional or technical terms whereas OT&E 
attempts to state requirements in operational terms (Appendix B) . Thus, 
their criteria thresholds are generally also different. Many times in 
DT&E/ tests concern only the weapon or a part thereof whereas in OT&E, 
tests primarily are concerned with the complete weapon system. Another 
difference is that DT&E often is testing and measuring a specific para- 
meter, holding other things constant to see the effect, whereas in 
OT&E many times no specific parameter is identified and only an opera- 
tional environment is created to see what the test results indicate. 
Thus, repeatability of tests is usually possible in DT&E whereas in 
OT&E it is not. Further, the people involved in DT&E are usually tech- 
nical whereas those in OT&E are not technical but operational. RAdm 
Monroe, former COMOPTEVFOR, addressed these differences in his comments 
about OT&E stating "that DT&E is a science and OT&E an art, and if 
there is duplication between the two, T&E is not being properly 
planned" [Ref. 13] . 

On paper, the success of a development program's T&E in the Navy 
rests with the Program Manager. He is responsible for the coordination 
of the various parts of the T&E effort, including the specific imple- 
mentation of the many facets of the DT&E program. The program's success 
is also dependent on the Program Manager's relationship and working 
procedures with OPTEVFOR, and on his interfaces with organizations 
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which have Navy T&E responsibilities. In the exercise of his respon- 
sibilities, the Program Manager must deal with the contractors, par- 
ticipating field activities, and various testing agencies to implement 
the DT&E program. He must operate within the matrix organization of 
the Navy for his technical assistance and support. And he must nego- 
tiate with OPTEVFOR to obtain a reasonable and affordable OT&E program, 
yet meet OT&E requirements. Those in the DT&E program are directly 
dependent on the Program Manager and OPTEVFOR is indirectly dependent 
on him as he provides the f lands to support both DT&E and most of OT&E. 

In addition the Program Manager must work with System Command and 
NAVMAT functional groups on test coordination and facilities planning, 
and his program must sustain the reviews and scrutinizing of CNO and 
DD ( T &E ) . 

Two especially difficult situations confront the Program Manager in 
his management responsibility and structure. First, it is the position 
of DD (T&E) that more value should be placed on OPTEVFOR information for 
milestone decision-making than on the developing agency* s data because 
OPTEVFOR is an independent test agency. DD (T&E) feels that the develop- 
ing agency has a vested interest in the program, therefore possesses an 
advocacy position. Thus DD (T&E) judges data received from DT&E in 
this light. Bowes made a similar observation in his thesis [Ref. 9] . 

Second, COMOPTEVFOR is double-hatted, heading-up OPTEVFOR and the 
T&E division in CNO (OP-983) . He also acts as the Assistant Director 
for OT&E (OP-098C) in support of the Director RDT&E in CNO, reporting 
from this position directly to the CNO. In the Program Manager's dis- 
cussions and negotiation with OPTEVFOR, the Program Manager is operating 
from a potentially difficult position. COMOPTEVFOR* s position in the 
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Navy organizational structure is very powerful compared to that of a 
Program Manager because of their relative positions with respect to 
CNO. 



C. TYPES AND CATEGORIES OF TESTS 

Figure 2 depicted the T&E types and categories of tests. Each of 
the T&E categories (phases) will be discussed separately as each one 
possesses unique objectives and features of T&E. Much of the informa- 
tion can be found in OPNAVINST 3960.10. It is presented here because 
it is continually referred to throughout this thesis. 

1. DT&E 

a. DT-I 

Development contractors and/or participating field activi- 
ties perform that DT&E possible at this time during the conceptual 
phase with experimental hardware to demonstrate concepts and feasible 
approaches to support the program validation decision (Milestone I) . 

b. DT-II 

Development contractors and/or participating field activi- 
ties conduct this DT&E during the validation phase, normally at the 
major component/sub- sus tern level, to support the full-scale develop- 
ment decision (Milestone II) . The hardware is usually of advanced 
development quality and its purpose is to demonstrate that design risks 
have been identified and minimized. 

c. DT-I II 

Development contractors design, build, and test engineering 
development, prototype, and pilot or limited production hardware. They 
essentially perform design evaluation testing, including redesign and 
re- test as necessary, to demonstrate compliance with contractual 



(specification) requirements. Testing areas include, as a minimum, 
performance , environmental , reliability /maintainability , compatibility , 
and safety. Tests are both sub-system and system level (ground, 
captive and free fight) with environmental evaluation tests, test 
analyze and fix, design proof tests, and qualification test programs. 
Reliability and maintainability demonstration programs are also 
generally conducted. 

The Program Manager may designate a participating field 
activity as the Technical Manager to provide necessary specialized 
technical and support engineering skills during this hardware develop- 
ment period. In this capacity the participating field activity may 
witness contractor performed DT&E and may choose to perform (repeat) 
certain tests judged to be critical to program success. In addition, 
certain components of ALGMs (e.g., warheads, fuzes, and propulsion 
systems) are designed, developed and tested by other participating 
field activities which have special competence, experience, and/or 
facilities necessary for conduct of these sometimes hazardous develop- 
ment efforts. 

Thus, contractually and/or through task assignments to 
participating field activities, design, analysis, test and evaluation 
proceeds through DT-III, including formal demonstration and qualifica- 
tion testing, sometimes referred to as Contractor Technical Evaluation 
or Contractor Demonstration Tests. The final sub-phase of DT-III for 
most ALGMs consists of subjection to a Navy Technical Evaluation, 
usually referred to as TECHEVAL. Its purpose is to provide evidence 
to the Program Manager for certification of readiness for OPEVAL 
(operational evaluation) . The TECHEVALs for ALGMs have historically 
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been performed for the Program Manager and NAVAIR by a field activity 
other than the one designated as Technical Manager. This has usually 
resulted in PMTC, Point Mugu performing this function. TECHEVALs per- 
formed by Point Mugu previously have placed significant emphasis on 
environmental testing in excess of ALGM specification limits and 
ground and airborne reliability testing to demonstrate compatibility 
with specification numeric and confidence level requirements. Compli- 
ance with maintainability requirements is also evaluated. 

DT-III then supports the first major production decision 
(Milestone III) and demonstrates that the ALGM design meets its speci- 
fications in performance, reliability, maintainability, supportability , 
survivability, system safety, and the totality of elements of electro- 
magnetic vulnerability. In addition, operability, human factors, 
logistics and system interface compatibility are evaluated, 
d. DT-IV 

This phase of DT&E is conducted after the first major pro- 
duction decision to verify that product improvements, or correction of 
design deficiencies discovered during OPEVAL, FOT&E, or fleet employ- 
ment, are effective. 

2. IOT&E 

a. OT-I 

IOT&E commences early in the program upon the request of 
the Program Manager with OT-I conducted by OPTEVFOR during the con- 
ceptual phase to support the program validation decision (Milestone I). 
Most acquisition programs do not require OT-I. However, this is the 
time for OT&E planning to take place. 
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b. OT-II 



OPTEVFOR conducts this phase of IOT&E during the validation 
phase to provide an early estimate of projected operational effective- 
ness and suitability of the ALGM, to estimate program progress and ini- 
tiate development of tactics and finally, to identify operational issues 
for consideration during the full-scale development phase (OT-III) . 
OPTEVFOR will participate in the ALGM captive flight test program to 
the extent practicable and may also monitor tests at contractor and par- 
ticipating field activities facilities to aid in performance of its 
early operational assessment, 
c. OT-III 

OPTEVFOR conducts this phase of IOT&E during full-scale de- 
velopment to support the program 1 s first major production decision 
(ililestone III) . OT-III concludes with a final sub-phase identified 
as OPEVAL which normally uses pilot production hardware and begins 
shortly after TECHEVAL or after incorporation of changes necessary to 
correct significant problems identified by TECHEVAL. Prior to OPEVAL 
and availability of pilot production hardware, OPTEVFOR will continue 
its IOT&E participation in the prototype ALGM evaluation of development 
hardware with captive flight tests and missile launches according to 
the program test plan. The purpose of OT-III and particularly OPEVAL 
is to evaluate ALGM operational effectiveness and suitability (includ- 
ing availability, compatibility, interoperability, reliability, main- 
tainability, human factors, logistic supportability and training 
requirements) and to continue tactics development. 
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3, FOT&E 



a, OT-IV 

OPTEVFOR may conduct this phase of FOT&E after the first 
major production decision but before production ALGMs are available if 
hardware problem fixes are to be evaluated, to continue tactics develop- 
ment, or to complete deferred or incomplete I OT&E. This phase may also 
be utilized for conduct of a fleet reliability evaluation program bv 
OPTEVFOR (i.e., early introduction and evaluation in actual fleet en- 
vironment) . 

b . OT- V 

OPTEVFOR conducts this FOT&E on production svstems as soon 
as they are available. Objectives include demonstration of the achieve- 
ment of program objectives for production system operational effective- 
ness and operational suitability, OT&E of the system in new environments, 
or in new applications, or against new threats. 

Figures 4 and 5 show the scope of DT&E and OT&E for ALGMs 
in a different format. Depicted are test categories and sub-categories 
for DT&E and OT&E, and typical phases in which the tests are performed. 

D. RESPONSIBILITY FOR TESTS 

DT&E is planned by, conducted by or for, monitored by, and reported 
by the developing agency. For ALGM programs, the Program Manager 
operates from the developing agency organization (NAVAIR) and delegates 
responsibility to support groups, field activities and contractors for 
various DT&E tests. The delegation vehicles are task assignments or 
contracts as appropriate and acceptance by the agencies constitutes 
obligation. 
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Figure 4. DT&E Tests 
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OT&E is planned by, conducted by or for, and reported directly to 
CNO by COMOPTEVFOR. With only a relatively modest number of personnel 
and resources, COMOPTEVFOR relies heavily on the facilities, resources, 
and personnel of the operating forces and government field activities 
for carrying out his mission. He exercises operational control over 
fleet units assigned for project support. Close liaison is authorized 
and exercised with appropriate elements of NAVMAT , the System Commands 
(HAVA I R for ALGM) and other T&E organizations to facilitate test support 
and information flow in carrying out assigned projects [Ref. 5] . 

E. CONDUCT OF TESTS FOR ALGMs 

For DT&E of ALGMs , tests are conducted by contractors, Navy Labora- 
tories, and participating field activities. OPTEVFOR is responsible 
for the conduct of the OT&E tests, which begin during the validation 
phase for most ALGM programs. Of importance to note is the planned 
overlap and concurrence of DT&E and OT&E throughout the acquisition 
period. 

The development contractors perform much of the DT&E testing. 

Several contractors are involved during the conceptual phase when the 
various concepts are evaluated. During the validation phase, the 
selected contractors conduct ground tests, and usually with government 
aid, conduct captive flight tests and flight test firings. During 
full-scale development, an ALGM development contractor conducts the 
DT&E he is contractually responsible for, which includes performance, 
environmental, reliability tests (as identified in section III C) at 
the sub-system and system level, on the ground and in captive flight 
tests and firings. Various contractors responsible for sub-systems 
of the missile system will perform similar applicable DT&E. 
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Navy development laboratories and other participating field activi- 
ties perform DT&E testing in ALGM programs as well. Development labora- 
tories support NAVAIR and the Program Manager in the early conceptual 
phase and usually continue in this capacity throughout the development 
cycle. During DT-I the laboratories conduct conceptual evaluations 
along with the contractors. During DT-II, they work with the selected 
contractors to demonstrate that design risks have been identified and 
the DT&E efforts support the Milestone II decision. Laboratories play 
an especially important role in the conduct of captive and free-flight 
tests. During DT-III they monitor much of the contractor testing and 
continue their role in captive and free-flight testing. They provide 
an assessment of the system and provide DT&E results to the Program 
Manager for his use concerning the production decision. Participating 
field activities perform DT&E for the Program Manager in certain sub- 
system areas which require special skills and/or facilities , namely in 
warhead, fuze and propulsion areas. PMTC, Point Mugu has historically 
conducted the TECHEVAL for ALGMs . 

OPTEVFOR begins conducting tests and performing evaluations of the 
ALGM data available during the validation phase's activities. This 
initial testing in OT-II is in conjunction with the DT&E program, con- 
ducting some of the captive flight tests and firings and monitoring the 
contractor, laboratory and participating field activity testing. This 
provides the necessary information for OPTEVFOR to make its projected 
operational effectiveness and suitability assessment for Milestone II. 
During full-scale development, OPTEVFOR continues to monitor the DT&E 
efforts and conducts many of the program's captive flights and missile 
firings. The OT&E efforts in this phase culminates in a formal OPEVAL 
which OPTEVFOR performs on the system forming the basis for its 
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Milestone III recommendation. OPTEVFOR also conducts or coordinates 
the operational testing associated with FOT&E. 

F. FACILITIES FOR T&E OF NAVY ALGMs 

Navy ALGM programs use six of the 26 activities designated elements 
of the DoD T&E Facility Base, also referred to as the Major Range and 
Test Facility (MRTFB) . The six are listed below. Associated mission 
statements as obtained from the Navy RDT&E Management Guide [Ref. 5] are 
given in Appendix C. 

1. Naval Ordnance Missile Test Facility, White Sands, New Mexico 

2. Naval Air Test Center, Patuxent River, Maryland 

3. Pacific Missile Test Center, Point Mugu, California 

4. Naval Weapons Center Ranges, China Lake, California 

5. Naval Weapons Laboratory Ranges, Dahlgren, Virginia 

6. Naval Explosive Ordnance Disposal Facility, Indian Head, Maryland 
The Program Manager and persons responsible for implementation of 

ALGM T &E projects can obtain the capabilities of these applicable T&E 
facilities by consulting references cited in Chapter 7, Section 073 of 
Ref. 1. The key factor in obtaining use of test ranges and other 
facilities is early contact with cognizant test facilities personnel. 
Early liaison will assist in the definition of a practical test plan 
to be incorporated in the TEMP, and will allow the facility the lead 
time required to provide the required support. 

G. FUNDING 

T&E funding responsibility rests with the Program Manager. The 
Program Manager must plan, program, budget, and fund the cost of all 
resources identified in the approved TEMP for all T&E from initial 
development through production, except for some fleet related expenses. 
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This includes all DT&E costs and practically all OT&E costs (OPNAVINST 
3960.10 cites the minor exceptions). Since the Program Manager must 
budget for the OT&E costs, it is essential that COMOPTEVFOR identify 
all requirements in sufficient time to integrate these into the program 
schedule and budget cycle [Ref. 4] . 

H. PLANNING 

OPNAVINST 3960.10 implements the planning policy of DoDD 5000.3 by 
specifying various requirements with regards to the TEMP . It notes 
that the TEMP defines the test and evaluation efforts for each acquisi- 
tion program and, as such, contains the integrated requirements of the 
developing agency for DT&E and COMOPTEVFOR for OT&E, their schedules 
and resource requirements. It further states that the TEMP will be pre- 
pared early in each new acquisition program, and approved prior to 
Milestone I; the TEMP will be prepared by the development agency with 
the OT&E portion prepared by COMOPTEVFOR; and that the development 
agency ensures the TEMP accurately reflects its planned approach to 
provide the necessary T&E to solve development design issues. Approval 
of the TEMP constitutes CNO direction to conduct the T&E program defined 
therein. An enclosure to OPNAVINST 3960.10 provides specific instruc- 
tions for TEMP preparation [Ref. 4] . 

In addition to the requirements of the TEMP, Navy T&E planning in- 
formation is contained in the Decision Coordinating Paper, which is 
prepared early in the system acquisition process. The Decision Coordi- 
nating Paper identifies the critical questions and areas of risk to be 
resolved by T&E, establishes the initial goals and thresholds, and also 
includes a summary statement of test objectives, schedules, and mile- 
stones. These are updated at subsequent revisions of the Decision 
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Coordinating Paper, and at the major milestone decision points 
[Ref. 6] . 

The Program Manager must plan his T&E efforts early because it con- 
stitutes a significant portion of his program budget. A representative 
budget must be submitted at Milestone I, therefore important and de- 
tailed program and T&E planning must occur during the conceptual phase. 
Plans summarized in the T&E section of the DCP lay the groundwork for 
earliest possible commencement of T&E and can influence significantly 
the program's budget. These planning inputs generally are prepared 
some time before the TEMP is assembled for Milestone I. TEMPs to date 
contain a level of detail and commitment which has caused them to be 
somewhat controversial and as a result agreement and approval do not 
occur until well into the development program. 

The budget process contains the impetus to accomplish early T&E 
planning. In addition, the Program Manager must realize that this early 
planning enables him to determine the scope, asset and resource require- 
ments of his T&E program. The early visibility will provide the Program 
Manager with valuable management insight into his planned T&E effort. 

I. NAVY T&E PROGRAM REVIEWS 

Aside from the program reviews cited in section II-E which are de- 
pendent on T&E results for basic inputs, program reviews of T&E are 
conducted periodically by the Program Manager, his T&E coordinator, or 
designee for T&E. These reviews, among other things, determine test 
program status, identify problems and determine corrective action status 
and responsibilities. Many times the T&E program reviews are a part of 
the system design reviews, as a broad spectrum of program personnel are 
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interested in the programs T&E activities. These reviews involve the 
participation of all T&E agencies responsible for ALGM T&E activities, 
including contractors, Navy laboratories, participating field activities 
and OPTEVFOR. However, OPTEVFOR's role in program reviews at the develop- 
ment level is limited, in keeping with its position as the independent 
test agency. T&E reviews help the T&E activities maintain the impetus 
and pace commensurate with the ALGM development progress. 

J. INFORMATION FLOW AND REVIEW 

T&E provides the information for decision-making. Whether it is de- 
sign evaluation data, operational utilization data, or risk assessment 
data, the T&E information must be disseminated to users to be useful. 

The Program Manager is aware of all T&E activity. All DT&E is authorized 
by the Program Manager and reported to his office. The only exception to 
this is when development contractors perform design tests as part of 
their internal development efforts; however, when the Navy requests such 
data, the contractor is usually cooperative in providing it. Although 
COMOPTEVFOR reports directly to CNO, COMOPTEVFOR provides significant 
OT&E test data and analysis to the Program Manager. Thus, through the 
program office the T&E information is available or provided directly to 
the necessary users. 

The TEMP has been a key document to help in the dissemination of T&E 
information. Its coordinated and integrated characteristic provides pro- 
gram personnel with visibility as to when tests are occurring, who is 
responsible, and provides other related test management information. 

This enables the T&E participants and users of the data to be involved 
in the testing, either actively or in a monitoring capacity, and to be 
aware of the availability of T&E information. 
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The Navy's philosophy of attempting to bring OPTEVFOR into the pro- 
gram's T&E efforts during DT&E helps greatly in an early assessment of 
the system and planning for OPEVAL. The results from DT&E and OT&E 
figure prominently in the development of the publications, manuals and 
training aids for the operating forces. This also pertains to the 
logisticians in their development and proofing of the Integrated Logis- 
tic Support Program. 
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IV. PROBLEMS IN THE MANAGEMENT OF T&E 
AND OF CONCERN TO ALGM PROGRAMS 



Problems will be addressed which have been identified throughout the 
T&E community and in ALGM programs, and of which the ALGM Program Manager 
should be aware. Some of the problems identified are of such a nature 
that their application can be to many program types or to T&E in general; 
however, all the problems alluded to should be a specific concern to an 
ALGM Program Manager, his designee for T&E, or any other individual re- 
sponsible for implementation of ALGM T&E. 

The problems are categorized into five areas: 

A. Planning 

B. Proliferation of Requirements and Reviews 

C. Resource and Schedule Constraints 

D. Redundant Testing 

E. Management of Diverse Activities and Functions 
A. PLANNING 

Probably the foremost problem relative to inadequate and inefficient 
T&E is poor planning by those responsible for performance and implemen- 
tation of the program. Planning problems encompass several dimensions, 
some of the important ones being: 

1. Inadequate and uncoordinated planning early in the development 
cycle and among the various responsible organizations 

2. Implementation of "success oriented" planning 

3. Lack of flexibility in T&E planning to accommodate problems and 
unknowns 
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4. Unrealistic requirements in the planning efforts 

5. Lack of design of tests, definition of test criteria, and know- 
ing the use of the test data 

6. Lack of definition of a mission profile 

7. Difficulty in converting operational requirements into test 
specification requirements 

8. Inability to convert specification performance requirements 
into T&E asset (resource) requirements 

These problems are discussed in the following sections, 

1. Inadequate and uncoordinated planning efforts early in the 

development cycle and among the various responsible organiza- 
tions 

Inadequate T&E planning has been the source of many difficulties, 
and has resulted in some cost and schedule growth in development programs. 
Planning simply was not being performed early enough in the development 
cycle, and when it was performed, many times it was incomplete, cover- 
ing primarily DT&E . This demonstrated the lack of the user's input. 

As a result initial program budget submittals at Milestone I were very 
deficient in the T&£ area. 

The requirement for a TEMP to exist at the major milestones 
(including Milestone I) for each program is an attempt to correct this 
problem. Unfortunately, many TEMPs do not exist until Milestone II, 
thus allowing the opportunity for many differences between what is pre- 
sented at Milestone I and Milestone II. In discussions with OPNAV 983, 
it was brought out that only a small percentage of program TEMPs are 
approved by Milestone II (a number as low as 10%) . This may be because 
the requirement for a TEMP is relatively new (since 1975); however, it 
may also be an example of people dragging their feet on implementing 
directives. There appears to be no sanction of the document and no 



penalty against programs which do not have a TEMP. Also, NAVAIR has 
yet to flow down implementation and guideline instructions to the upper 
level directives. 

Many times the T&E program plans submitted were not coordinated 
between the various T&E organizations and responsible program people. 

This situation may exist when a program does not have a T&E coordinator 
or the Program Manager allows disjointed efforts to exist. Adequate 
early planning would minimize this situation. Again, the requirement 
for a TEMP is an attempt to obtain a coordinated T&E program. A coordi- 
nated program can also help to reduce another major problem, redundant 
testing. 

The Defense Science Board, in the reports of its task force on 
T &E, reported an indication of widespread inadequate early planning for 
T£E in past programs [Refs. 14 and 15]. 

2 . Implementation of "success-oriented" planning 

When test planning is performed, it generally includes all known 
testing requirements and contains some integration of the testing activ- 
ities. However, the planning is basically "success-oriented”, whereby 
test failures with corresponding contingencies are not included in the 
plans. Only in flight testing are back-up tests consistently identified, 
although many times insufficient. Ground testing, especially in DT&E, 
suffers from this characteristic. The Defense Science Board in its most 
recent review on T&E also stated that success-oriented planning was 
performed [Ref. 15]. 

3 . Lack of flexibility in T&E planning to accommodate problems and 

unknowns 

Many times the test program planned did not allow sufficiently 
for problem solving and unknown situations which surface during 
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development, contributing to schedule stretchout and increased cost in 
test programs. This is a very difficult feature to plan for, but one 
to which the Program Manager should give more attention in future plan- 
ning. Many times problems are uncovered during testing, or some 
unknown occurs, which necessitate changes in the course of action. How- 
ever, a real difficulty exists in that many times program funds and 
schedules do not possess the flexibility required to make the necessary 
program adjustments. This situation was highlighted by the Defense 
Science Board [Refs. 14 and 15] • In discussions they had with industry 
representatives, industry indicated almost universally an erosion process 
of program contingency funds throughout the bidding and negotiation 
process . 

4 . Unrealistic requirements in the planning effort 

Since the implementation of DoDD 5000.3 in 1973, improved plan- 
ning has taken place in T&E . The OT&E people of COMOPTEVFOR were given 
a strengthened charter which they have proceeded to implement. However, 
the organizations responsible for DT&E have existing "standard operating 
procedures" which they implement. Combining these two perceptions into 
the test program plans resulted in an abundance of testing being speci- 
fied, with high cost T&E programs. During this period, studies were 
reporting that inadequate and insufficient T&E was being performed. 

Future planning must look for most cost-effective T&E programs. 

5 . Lack of design of tests, definition of test criteria, and know- 
ing the use of the test data 

Poor planning in the design of tests, definition of test 
criteria, and in not knowing the expected use of the test data results 
in proliferation of testing, as well as wasteful testing and testing 
with questionable results. Such a situation should not exist in today's 
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limited funding environment. Testing, especially flight testing, is 
very expensive. This problem includes both DT&E and OT&E. The OT&E 
people must be especially concerned with this situation because of 
their desire to test in the operational environment which is more 
difficult and costly. 

The Defense Science Board report contains several guidelines 
relative to design of tests which are appropriate to consider. They 
include: 

a. "Ensure that the whole system, including the user people, 
is tested. Realistically test the complete system, in- 
cluding hardware, software, people and interfaces. 

b. Ascertain that sufficient time and test articles are 
planned. When the technology is stressed, the higher 
risks require more test articles and time. 

c. In general, parts, sub-systems and systems should be 
proven in that order before incorporating them into the 
next higher assembly for more complete tests. 

d. Major tests should never be repeated without an analysis 
of failures and corrective action" [Ref. 7] . 

6 . Lack of definition of a mission profile 



In recent years, the value of a mission profile or the several 
mission profiles of a weapon system has been realized and become a re- 
quirement for programs. This resulted from a weakness in the user-pro- 
ducer dialogue. The operational requirements of the user were not get- 
ting into the development specifications- The DSB refers to this 
recommending that specification requirements be stated in functional 
terms rather than design values, necessitating strong user involvement 
[Ref. 14]. 

Specifying a system's mission profile requires knowledge of the 
operational performance requirements, and incorporates a realism into 
requirements and subsequent testing. Obtaining a system's mission 
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profile requires planned and coordinated efforts by the program's using, 
development, and testing teams. The lack of this in the past has con- 
tributed to wasteful, inefficient and less than desirable T&E efforts. 

7. Difficulty in converting operational requirements into test 
specification requirements 

This has been a problem for many years and probably will continue 
to be a prime area for improvement in the future. Better early planning 
can minimize this difficulty, along with an improved user-producer dia- 
logue. During the generation of the system's performance and design 
specifications, operational requirements should be the source for these 
specifications, with strong user involvement. At the same time, the user- 
producer dialogue must be extended to include test program planning with 
the specification generation task. Only if this is done will the DT&E 
and OT&E programs be testing real operational or operational related re- 
quirements. The Defense Science Board in its most recent study cited 
inadequate requirements statements as a problem [Ref. 15] . 

Currently OPTEVFOR is closest to the operational community? 
however, in many instances its personnel lack expertise to translate the 
perceived operational requirements into development specification 
language. Also, 0P1JAV, which is a user organization, appears to be often 
influenced by technical inputs from contractors and development labora- 
tories. Thus, the operational requirements may be losing user represen- 
tation. 

8 . Inability to convert specification performance requirements into 
T&E asset (resource) requirements 

When the test agencies have made their inputs, and the test per- 
formance requirements have been established, the problem exists of con- 
verting these into resource requirements. It is important to perform a 
requirements analysis in conjunction with the design of the test to 
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determine the asset limitations, both from feasibility and affordability 
points of view. The testing required to perform the desired program is 
simply resource limited. Among resource considerations are such items 
as facilities, personnel, test hardware, threat simulation, and support 
equipment. Both the NMARC study [Ref. 16] and Defense Science Board re- 
port [Ref. 14] identified this limitation in Navy T&E . Test resources 
are not keeping pace with development program requirements. 

In general, the Navy must continue to make improvements in its 
T&E planning. With progress being made through the implementation of 
DoDD 5000.3 and OPNAVINST 3960.10, and the subsequent TEMP, plans for 
T&E should be improved. However, this result is not yet clearly apparent. 
.Much work is being expended, supported by the fact that many TEMPs are 
in progress or draft form, but only a small percentage are formally 
approved. Additional efforts to perform planning earlier in the devel- 
opment phases would prove to be beneficial. In addition to providing 
the Program Manager with valuable management information, such early 
planning could be used as a "strawman" for the TEMP, thus possibly re- 
ducing its generation and approval cycle time. This early planning 
would also highlight any potential T&E gaps which impact on program 
milestone decisions (T&E gaps are breaks in testing due to hardware un- 
availability, anywhere during the development and production cycle, not 
just between IOT&E and FOT&E as used in Ref. 14) . This early planning 
must also involve very strongly the user and OPTEVFOR. 

B. PROLIFERATION OF REQUIREMENTS AND REVIEWS 

During the performance of T&E throughout the development cycle, new 
requirements surface and are imposed on existing programs as well as 
new ones. These requirements are not planned for because they did not 
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exist at the time of planning. Such requirements might be in the form 
of AR's (Aeronautical Requirements) which are essentially new require- 
ments which generally affect testing procedures. Other requirements 
might be program reviews which were not anticipated and may reflect a 
current fleet concern or a shift in emphasis in certain disciplines 
(e.g., reliability). Examples of such reviews at the Washington level 
were the developing agency special management reviews addressing heavily 
the reliability concern, and the production reliability design review 
[Refs. 17 and 18]. Progress on the development program and in T&E is 
affected by such reviews because of the support required to prepare and 
perform the reviews. Reviews such as these are not provided for in 
program budgets and contribute to a depletion of the management discre- 
tionary funds. Review requirements such as these also seem to come and 
go, depending on the people in the review chair or those in power. The 
N21ARC report cited the difficulty the Program Manager has responding to 
all reviewers and requirements from the many layers of people in the 
Navy organization [Ref. 16] . 

There are other reviews which impact on T&E as well. Reviews at 
participating field activities such as design reviews, environmental and 
qualification program reviews, and those related to range safety require 
special efforts from the T&E organization and many times are beyond 
that planned in the budget. Proper attention to such requirements in 
early planning could minimize this impact. However, a proliferation of 
these requirements may be detrimental to efficient development progress. 

Unique capabilities may fall into the category of proliferation. 

For example, some test activities may have built up capabilities over 
the years, both in facilities and manpower, which are quite special, 



and these activities tend to perpetuate the use of such capabilities 
and their existence. Capabilities such as PMTC , Point Mugu's acoustic 
vibration facility and Naval Explosive Ordnance Disposal Facility (NEODF) 
Indian Head's propellant manufacturing and testing capability fall into 
this category. Although these facilities and capabilities are important 
ones, they may not fit into every program’s T&E plans. However, once 
activities possess a unique capability, the activities are determined 
to keep it in use. Thus, requirements may be imposed on the program to 
include the use of these special test facilities as part of the test pro- 
gram. The more logical approach is for the program to determine what 
testing is required and then determine the best method and place to 
accomplish that testing. 

C. RESOURCE AND SCHEDULE CONSTRAINTS 

Resource and schedule constraints add to the complexity of accomplish- 
ing the T&E assignment. These constraints have been used as an excuse 
for non-compliance with directives and instructions applicable to defense 
development programs [Ref. 14] . 

The primary resources for T&E are personnel, facilities and equipment. 
Talented T&E people are key to performing the T&E function. T&E suffers 
from a lack of talented people and this area must be made more attractive 
and rewarding. Facilities, ranges, instrumentation and support equip- 
ment are growing obsolete and have maintenance problems. With the de- 
crease of defense funding in the 1970's, less flight testing is performed 
and fewer dollars are available to maintain ranges and facilities at 
acceptable levels. In addition, there is a need to incorporate improve- 
ments necessary to keep pace with the newer, more sophisticated missile 
systems currently being developed. Support equipment is often overlooked 
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in the choice for new targets , range upgrading or repair, but equipment 
such as range trackers and telemetry systems are equally important in 
achieving satisfactory T&E . The NMARC report, the Defense Science Board 
reports and discussions with the NWC, China Lake T&E facility manager 
all highlighted this problem relating to T&E resources [Refs* 14, 15, 
and 16] . 

A recent concern related to maintaining adequate test ranges has 
been encroachment by civil activities on the government’s vast range 
land and the sea and airspace needed for testing. Such a movement is 
a potential threat to this test resource. Lt Gen W. E. Lotz (Ret) , 
head of DD (T&E) , cited this problem as very real in his presentation to 
Congress on defense T&E [Ref. 3] . 

Schedule constraints may be the single greatest excuse given for 
not accomplishing the T&E plans or why T&E is performed out of sequence 
with program milestones. Usually programs are striving to meet or better 
a schedule, but many times there is some program slippage. When program 
decision points are based on a predetermined schedule, many times the 
decision milestone takes place and the following phase is initiated 
before the T&E is completed, because T&E characteristically is more 
prevalent in the latter part of the development phase. The report of 
the Commission on Government Procurement cited the existence of this 
situation and recommended instead that milestone decisions be based on 
program information and T&E results [Ref. 11] . The Defense Science 
Board also noted this situation and made a similar comment [Ref. 15] . 
However, schedule constraints continue to prevail because people in 
command are uncomfortable with flexible milestones and the budget cycle 
operates from a fixed schedule base. 
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DD (T&E) does make a major recommendation to DDR&E at the program 
milestone reviews. However, rarely has a program been stopped because 
of insufficient T&E. This occurs because there are very few yes and 
no answers in T&E, creating a wide zone of uncertainty, risk and judge- 
ment evaluation. But, as the Defense Science Board reported, DD (T&E) 
and the milestone decision makers have been unable to make risk assess- 
ments, and the program has moved on according to the calendar driven 
schedule [Ref. 15] . Management must better address this situation, 
develop new plans and procedures, and then adhere to performance driven 
milestones. The latest directives relative to system acquisition, 
currently in review, are attempting to mandate performance driven mile- 
stones rather than schedule driven milestones. 

D. REDUNDANT TESTING 

Redundant testing is testing more than is necessary through duplica- 
tive efforts among various test agencies. The potential for increased 
redundant testing has occurred since the issuance of the new directives 
in the 1970s with their added emphasis on test and evaluation. These 
new directives, as applied to the Navy, gave OPTEVFOR additional respon- 
sibility and stature, resulting in an expansion of OT&E testing require- 
ments. The requirements of DT&E have remained the same with some 
inherent redundant testing built into the development program as the 
government provided a "watchdog" function on the contractor. The com- 
bined effect of the CT&E and DT&E requirements results in additional 
duplication of tests. Prime examples of this situation in ALGM programs 
are in environmental testing, reliability testing, and captive flight 
testing. 
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In environmental testing the development contractor is required by 
his contract to perform environmental tests on the system and applicable 
sub-systems. The navy then proceeds to repeat much of the testing dur- 
ing its TECHEVAL at PMTC, Point Mugu. This is classified as confirma- 
tion of the contractor’s tests. In the past, much of this type of test- 
ing was performed to prepare or train Point Mugu for its role in fleet 
related testing after IOC. Such redundant testing simply depletes the 
already strained RDT&E budget. 

The situation in reliability testing could become quite serious. 

There has been much emphasis on reliability in the past several years 
because of high maintenance costs and fleet disappointment in weapon 
systems that did not perform at all or to their advertised capability. 

As a result, many T&E organizations are proposing their own reliability 
test and demonstration program for the same weapon system. Also, many 
people in program review positions are encouraging and/or requiring the 
demonstration test programs. However, the fact must not be overlooked 
that such testing is quite expensive and a large amount of testing is 
required for any one organization to be able to state a reliability 
from a demonstration program with any high level of confidence. For 
example, if a missile has a reliability requirement of 75 hrs MTBF , 700 
hrs of operating time with no more than 5 failures are required to 
demonstrate at the 90% confidence level the specified reliability require 
ment (based on the exponential distribution assumption) . The contractor 
is contractually required to demonstrate the reliability of the system. 
That alone requires a substantial amount of testing. A requirement for 
TECHEVAL people and the OT&E people also to perform reliability demonstra 
tion test programs is often uneconomical and redundant. Each 
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organization would claim, though, that its demonstration program was 
different because of environmental changes. 

In captive flight testing of ALGMs , tests are performed by the de- 
velopment agency (contractor and Navy development laboratory) as part 
of DT&E , the TECHEVAL agency, also as part of DT&E , and OPTEVFOR as 
part of OT&E . Although some unique tests are performed by each test 
agency, there is some duplication as well. Claims of program advocacy 
by the development agency and testing by " independent" test agencies 
are reasons used to justify such redundant testing. 

The question of redundant testing must be examined carefully before 
implementation because of the tight fiscal situation of present times. 

It appears reasonable that through proper planning of the development 
contractors test program and associated monitoring by the Navy, some 
redundant testing could be eliminated. In addressing the problem of 
redundant testing, the objective should not be to remove all duplica- 
tion. It should ensure that where such duplication exists, it is 
visible, affordable, controlled, purposeful, and contributes appro- 
priately to the T&E program. The Defense Science Board noted that there 
appeared to be little or no overtesting done under the current direc- 
tives, but some redundant testing occurred [Ref. 15]. 

E. MANAGEMENT OF DIVERSE ACTIVITIES AND FUNCTIONS 

One of the significant difficulties which is often overlooked in 
managing T&E is that the Program Manager and T&E coordinator must deal 
with many diverse activities and functions. In addition to DD(T&E) in 
OSD, OPTEVFOR and OP-983 in OPNAV, the T&E manager must operate with 
the matrix organizational structure of the Navy for his functional T&E 
and technical support. Many of the T&E activities and organizations 
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within the Navy possess their own standard operating procedures and ex- 
hibit various forms of parochialism. Dealing with this characteristic 
requires special talents and capabilities by the T&E manager in order to 
operate effectively. The important point is to realize everyone* s posi- 
tion when the T&E planning takes place. 

Management of the Navy and development contractor activities simul- 
taneously, their interfaces and interfacing, is another formidable task 
for the ALGM Program Manager. Many times competition and adversary rela- 
tionships exist between the two groups. In T&E it is often the Navy*s 
position that the contractor has something to gain by taking advantage 
of the Navy and will do so in its testing program. There is disagree- 
ment between the two organizations performing the tests partly because 
the contractor feels it's his responsibility and the Navy feels it must 
perform the tests to validate the results. Much of the current thinking 
in ALGM programs is that the system development contractor is responsible 
for all the development activities, including the many aspects of T&E 
from component and assembly testing to system flight testing. This 
creates a management problem in that the Navy is responsible for the 
aircraft and test ranges where the system flight testing is performed. 

In summary, it is necessary for the ALGM Program Manager to be con- 
scious of each problem in order for him to better manage and implement 
a successful and effective T&E effort. Sometimes the problem situation 
is of such a nature that the ALGM Program Manager has little opportunity 
to exert influence on or correct the ultimate outcome; however, simply 
realizing the problem exists helps him tolerate the situation. Never- 
theless, many of the T&E management problems can be dealt with directly 
by the ALGM Program Manager. 
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The T&E management problems identified in this section will be 
addressed in the following discussion of potential alternatives for 
performing an ALGM program* s T&E . Some of the problems will be more 
prevalent in one alternative than in another. 
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v. AIR-LAUNCHED GUIDED MISSILE T&E PROGRAM 
MANAGEMENT ALTERNATIVE APPROACHES 



The Program Manager has several basic approaches or alternatives 
with respect to implementation of an "adequate" T&E Program for an ALGM. 
Four alternatives will be considered in this section, all of which are 
basically compatible with the requirements and guidelines of DoDD 5000.3 
and OPNAVINST 3960.10. The four alternatives are: 

A. Separate Test Programs in Series 
3. Combined Test Programs 

C. Integrated Test Programs 

D. System Level Functional Test Program 

Each alternative when analyzed suffers to a certain extent from the T&E 
management problems identified in section IV. The applicability of 
those management problems and other problems to a particular alternative 
is discussed in this section. The good features of each approach are 
also discussed. 

A. SEPARATE TEST PROGRAMS IN SERIES 
I . Management Approach 

The Separate Test Programs in Series management approach adheres 
rigidly to the requirements and guidelines of OPNAVINST 3960.10 and is 
a means of implementing DT&E and OT&E efforts similar to those described 
in section IIIc. The Program Manager attempts to structure the T&E 
program utilizing contractors, participating field activities, and 
support activities. Each program T&E participant will advocate and 
attempt to conduct its own test program to demonstrate ALGM specification 



61 



and or DCP compliance. The nature and degree of independence of each 
participant will be reflected in the structured inputs to the TEMP 
preparation and review cycle resulting in basically three separate 
test programs being conducted as follows: 

a. Contractor Program 

DT-I through DT-III as described in section III C 1 culmi- 
nating with Contractor Demonstration Tests, including: 

(1) Part and component qualification, assembly, and sub- 
system tests (accelerated life, overstress, and design 
evaluation and qualification) , sub-system and section 
integration and compatibility tests. 

(2) ALGM system level performance tests both on the ground 
(chamber tests) , on board the aircraft, and in the 
captive carry mode concluding with several missile 
launches from aircraft, 

(3) reliability demonstration for compatibility with 
specification requirements (ground and airborne) 
testing for captive carry MTBF , launch from aircraft 
and free flight to target reliability) at the required 
confidence levels , 

(4) maintainability demonstration for compatibility with 
specification requirements, 

(5) environmental and safety qualification (including 
electromagnetic compatibility) to verify compliance 
with specification design requirements. 

b. TECHEVAL Program 

Performed during che final sub-phase of DT-III and 

includes : 
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(1) ALGM system level performance both on the ground 
(chamber tests) , on board the aircraft, and in the 
captive carry mode concluding with several missile 
launches from aircraft, 

(2) reliability demonstration for compatibility with 
specification requirements (ground and airborne testing 
for captive carry MTBF, launch and free flight to 
target reliability) at the specification required 
confidence levels, 

(3) maintainability evaluation, 

(4) environmental tests to verify compliance with speci- 
fication design requirements. 

Occasionally, the TECHEVAL environmental tests have included overstress 
tests to evaluate hardware design margins, 
c. OT&E Program 

OT-II and OT-III as described in section III C 2 culminating 
with OPEVAL, which includes: 

(1) ALGM system level performance tests on board the air- 
craft, on the ground and in the captive carry mode 
concluding with several missile launches from the 
aircraft, 

(2) airborne evaluation of captive carry MTBF and free 
flight to target reliability for compliance with the 
ALGM system specification confidence levels, 

(3) evaluation of ALGM system maintainability 
characteristics . 

Other operational suitability characteristics of the ALGM evaluated 
are: compatibility, interoperability including human factors, and 
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logistic and training adequacy. Additionally, OPTEVFOR will continue 
tactics development and refinement, 
d. FOT&E Program 

OT-IV, as described in section IIIC3, may be conducted by 
OPTEVFOR but is rarely defined and/or planned for early in the program's 
life. FOT&E is beyond the scope of this thesis. 

2 . Applicable T&E Management Problems 

The principal Developing Agency for Navy ALGM's is the Naval Air 
Systems Command and AIR-06 is its responsible T&E agency (Figure 6) . 
AIR-06 has been in existence for approximately three years and has not 
as yet promulgated T&E Instructions or Guidelines in response to 
OPNAVINST 3960.10. This has contributed to something less than full 
implementation of DoDD 5000.3 and OPNAVINST 3960.10 T&E requirements 
and permits circumstances such as the Separate Test Programs in series 
to occur too easily rather than explore a more efficient process. 
According to AIR-06 personnel, it has been very difficult for AIR-06 to 
effectively impact ALGM T&E programs and to fully and efficiently im- 
plement the TEMP process because of the lack of published T&E instruc- 
tions. A further constraint to date on AIR-06 ' s effectiveness is the 
fact that its personnel have been utilized for the most part only in an 
advisory capacity by ALGM Program Managers. Thus, without more specific 
guidance and direction from within NAVAIR, TEMP preparation has been 
dragging and ALGM T&E management problems similar to those described in 
section IV continue. T&E management problems arising from the Separate 
Test Programs in Series method of operation are as follows: 
a. Planning 

As previously indicated in section IVA.l, NAVAIR' s current 
position with respect to flowdown of T&E implementation and guideline 
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Figure 6. Navy Responsible ALGM T&E Agencies 




instructions, makes it difficult to influence or even impress upon ALGM 
Program Managers the necessity for early, thorough T&E planning. Further 
complicating this situation, is the fact that DD (T&E) , OP-983C, and NAVMAT 
have all indicated, when interviewed by the authors, that a strawman TEMP 
is all that is required and reasonable to expect for Milestone I. It was 
stated also that OPTEVFOR was usually not available for early planning 
efforts prior to the Milestone I TEMP effort because of its heavy involve- 
ment in OPEVALs . A contradiction exists in that OPTEVFOR personnel, when 
interviewed, indicated problems with early planning prior to Milestone I 
were due to reluctance on the part of Program Managers to commit to 
supporting a detailed T&E program. OPTEVFOR further stated that there 
were no real sanctions imposed on the Program Managers or their programs 
if they did not do the necessary thorough investigation and planning 
effort. Too often, Program Managers are unable or unwilling, because 
of the myriad of tasks in the early stages of the program, to come to 
grips with the T&E details and to challenge or even question established 
precedents or standard operating procedures. Under these circumstances, 
T&E programs like the Separate Test Programs in series can result. Once 
brought to life at Milestone I, the drain on scarce program resources 
commences and is extremely difficult to curb and control, 
b. Proliferation of Requirements 

With the Separate Test Programs in series approach the Pro- 
gram Manager is much more likely to be confronted with proliferating 
requirements which impinge on scarce program resources. ACAT I or II 
programs usually have many diverse participants (participating field 
activities, such as : PMTC, NWC, NWL) who have unique internal require- 
ments which must be satisfied during the life of the program. These re- 
quirements may take the form of management review, special technical 
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reviews, special test programs or regimes, and additional requirements 
designed to perpetuate utilization of unique facilities or capabilities. 
For example, NWC, which has functioned as Technical Manager for NAVAIR 
on several ALGM programs and as a participating field activity develop- 
ing major components on other ALGM programs, has internal instructions 
which require extensive management reviews (e.g., Design Review Committee 
Environmental Qualification Review Panel, Ammunition Safety Committee) . 
Any or all of these retain the authority to impose requirements for 
additional program resources, i.e., additional ALGM assets for testing 
(larger quantities), additional tests or sequences which may require 
different or enhanced facilities or more schedule time. 

TECHEVAL's conducted at PMTC invariably "require" subjection 
of ALGM ' s to extensive environmental tests not only to verify compliance 
with specification design requirements but also to evaluate design 
margins (i.e., overstress or test to failure type of tests). PMTC also 
is a proponent of acoustic vibration chamber testing and possesses one 
of the few facilities of this type capable of testing all-up-round ALGM's 
Consequently, ALGM ground reliability tests (simulating the captive 
flight environment) are conducted to demonstrate the specified MTBF at 
the required confidence level using this chamber. This is required 
even though the ALGM contractor has already performed a similar demonstra 
tion to satisfy contractual provisions under a government approved pro- 
cedure with government monitoring. 

The Program Managers are often faced with high level manage- 
ment review similar to those reported by NMARC [Ref. 16] . The reviews 
utilize T&E data and are usually conducted at such a high level (Flag 
Rank) that the program essentially comes to a halt while presentations 
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are prepared, participants assemble, and the review occurs. Quite com- 
monly, this occurs several times during ALGM development and impacts 
significantly on program resources. 

With the increased emphasis on IOT&E, OPTEVFOR is involved 
in monitoring DT&E activities. In addition to operational testing, 
OPTEVFOR personnel have said in recent interviews that they may choose 
to repeat DT&E testing where apparent weaknesses were revealed, 
c. Resource and Schedule Constraints 

The Separate Test Programs in Series approach can present 
especially significant resource and schedule constraint problems for 
the Program Manager or his designee for T&E . Inadequate early coordi- 
nated planning impacts severely on lead times necessary to acquire 
requisite facilities and instrumentation with appropriate staffs which 
may be necessary to support new ALGM technology. A further problem 
aggravating this situation is the fact that the Navy's Test and Evalua- 
tion Facility base is still loosely knit and without strong central 
coordination. NMARC [Ref. 16] commented on this problem and made 
several recommendations which should improve the situation as NAVAIRs 
relatively new T&E Directorate, (AIR-06), assumes its leadership role. 

With the high possibility for duplicative testing under 
the Separate Test Programs in Series approach, problems like facility 
scheduling and allocation of priorities, become quite significant. 
OPTEVFOR possesses no facilities, instrumentation, or ranges of its own 
and, because of the nature of some ALGM's, must rely on DT&E resources 
to conduct tests for it. Certain range and instrumentation capabilities 
available only at government facilities are required throughout the 
ALGM development cycle and present a constant scheduling problem because 
of contractor, Navy lab, and OPTEVFOR requirements. 
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Another aspect of the resource and schedule constraints 
related to facilities, instrumentation, and ranges is the impact of 
obsolescence and increased maintenance costs. The Navy's facilities 
are having a difficult time keeping pace with technological advances 
in weaponry and expanded emphasis on testing is requiring more main- 
tenance on existing facilities and instrumentation. This was confirmed 
in a recent interview with the Head of the NWC T&E organization, where 
inany ALGM tests are conducted, 

d. Redundant Testing 

Contractor contractually required Reliability Demonstration 
(both ground and airborne) and Environmental Qualficiation tests are 
repeated during ALGM TECHEVAL. OPTEVFOR conducts what amounts to an 
airborne Reliability Demonstration [Ref. 19] during its evaluation of 
ALGM operational effectiveness and suitability. The redundancy problem 
occurs because the agencies cited feel that must repeat or independently 
conduct reliability test programs to verify compliance with specifi- 
cation minimum acceptable MTBFs or probabilities of success at the 
stated confidence levels even though the contractor may have success- 
fully done so. This can require an extraordinary amount of time 
(schedule) , a significant commitment of aircraft and manpower, and a 
very large number of ALGM test assets. 

e. Management of Diverse Activities and Functions 

The Separate Test Programs in Series approach severely 
taxes the matrix management approach which ALGM Program Managers are 
compelled to use as a matter of policy. In the current environment 
of the shrinking dollar and fewer new programs, competition between 
field activities for T&E work is strong and the feeling of the need 
to perpetuate activities and capabilities is pervasive throughout the 
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Navy's T&E community. The competition between Navy activities within 
DoD's Major Range and Test Facility Base (e.g., PMTC, NATC, and NWC) 
creates borderline adversary relationships that require careful 
handling on the part of the Program Manager or T&E coordinator to 
assure successful , timely execution of the T&E program. 

Another facet of this competitive situation is that DD (T&E ) 
is actively pursuing reduction of unwarranted duplication of facilities 
and functions among facilities which constitute the Major Range and 
Test Facility Base ( MRTFB ) [Ref, 3] , 

3 . Other Problems 

There are several problems outside the five basic areas identi- 
fied in section IVA.l which also impact the Program Manager in his 
attempt at planning and implementing an adequate T&E program using 
the Separate Test Programs in Series approach. These other problems 
influence the T&E program as follows: 

a. If the Separate Test Programs in Series approach is followed, 
with little or no interplay between DT&E and OT&E while the program is 
stretched out, appropriate attention may not be directed toward main- 
tenance of the capability to meet the threat. The ALGM may satisfy 

the specification requirements but when subjected to more stringent 
OT&E tests, particularly OPEVAL, may be found inadequate to meet the 
evolving threat. 

b. With this approach (Separate Test Programs in Series) , 
stretchout of the schedule and consequent dollar impact means that 
there will be a great deal of pressure on OPTEVFOR to expedite OPEVAL 
because of the rapidly approaching IOC date. There will be entreaties 
to compromise test objectives to conserve assets and shorten the time 
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frame and somehow soften the dollar crunch. This same situation will 
also make it extremely difficult to plan and execute the FOT&E effort. 

c. ALGM specifications are still stated in terras of engineer- 
ing design requirements and performance parameters measurable in the 
laboratory which may not be relevant to an operational commander, 
according to OPTEVFOR representatives. Without specifications stated 
in operational terms, it is difficult for OPTEVFOR to translate speci- 
fication requirements into operational requirements and evaluate test 
results accordingly. The Commission on Government Procurement [Ref. 11] 
observed this same problem and suggested that once the engineering 
specifications are validated in the DT&E effort, the IOT&E effort should 
be directed toward determining whether they have operational value or 
not. 

d. According to a DD(T&E) staff member when interviewed re- 
cently, DD (T&E ) places little reliance on DT&E test data, including 
TECHEVALs, because of the lack of independence from the Developing 
Agency and because of contractor's and the Program Manager's program 
advocacies reflected in their interpretation of test data and presenta- 
tion of results; i.e., need to sell the program. They (DD(TSE)) feel 
that OPTEVFOR is objective and unbiased and that operational testing 

is the prime valid measure of an ALGM 1 s worth and utility and this en- 
ables them to provide the DSARC with objective unbiased data for the 
decision making process. DD (T&E) feels however that OPTEVFOR needs 
more analytical and technical expertise, citing their past difficulty 
in overall test planning and design of experiment and pointing to a 
future need for the ability to perform simulations which are going to 
have to play a larger role in future T&E programs because of escalat- 
ing ALGM asset costs and test costs. 
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4. Good Features 



Good features of the Separate Test Programs in Series approach 
are as follows: 

a. The competition created between T&E activities within the 
Navy T&E community, while hard on the personnel involved, may yield a 
more efficient, technically competent Major Range and Test Facility 
Base, because it may at least force internal self-examination of capa- 
bilities, costs, and personnel, leading to reduction of duplication. 

b. DD (T&E) places more emphasis and reliance on operational 
test results; therefore, with the potential for more data regarding the 
ALGM's effectiveness and suitability, the DSARC decision process may be 
easier with less risk. The other potential attribute is that more time 
is available for proofing and debugging software so that it is possibly 
less of a problem when initially deployed. 

3. COMBINED TESTING 

DoDD 5000.3 states that, "development testing and early phases of 
operational testing may be combined where separation would cause delay 
involving unacceptable military risk, or would cause an unacceptable 
increase in the acquisition cost of the system" [Ref. 6] . 

1. Management Approach 

The latter portion of the quote from DoDD 5000.3 would seem to 
provide some relief to the Program Manager when faced with inputs to 
the TEMP from the T&E participants which appear similar to those dis- 
cussed under Separate Test Programs in Series; i.e., separately con- 
ducted duplicative and redundant test programs that can cause an unac- 
ceptable increase in the ALGM* s acquisition cost. If this problem is 
recognized early enough in the T&E planning stages, a viable management 
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alternative for the Program Manager is to strive for a Combined Test- 
ing approach through the medium of the TEMP preparation and review 
cycle. This discussion assumes the Program Manager is faced with just 
such a situation, recognizes it sometime prior to Milestone II and 
attempts to rectify the projected schedule and cost growth through 
combining elements of DT&E and OT&E. The proposed elements of the 
TEMP (i.e., Contractor Test Program, TECHEVAL Program, and OT&E Program) 
are essentially the same as presented in the preceding section VI A. 

OPNAVINST 3960.10 permits combined testing but states that, " the 
final sub-phases of DT-III and OT-III (TECHEVAL and OPEVAL) will not 
normally be combined". While this appears to be a significant hindrance, 
the major obstacle for the Program Manager to overcome is the fact that 
he and COMOPTEVFOR are jointly responsible for preparation of the TEMP 
and if they cannot achieve agreement the problem is passed to the OPNAV 
program sponsor via OP-983. The Program Manager states the areas of 
disagreement when the TEMP is submitted and COMOPTEVFOR responds, in 
writing, with the rationale for those areas where there appears to be 
disagreement. Due to the Navy's command structure, with COMOPTEVFOR 
being double-hatted and filling the position of DCNO for T&E , the Pro- 
gram Manager would appear to be in a disadvantageous position when 
taking exception to the OT&E program. 

TECHEVAL is an equally difficult area for the Program Manager 
to modify or waive because of NAVAIR standard operating procedures. 

With few exceptions, ALGM Program Managers find it almost impossible 
to modify the approach to ALGM TECHEVAL historically taken by PMTC, 

Point Mugu, because the NAVAIR T&E community looks upon this as its 
"independent" evaluation of the system. 
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The most likely candidates for combining tests appear to be 
those ALGM system level tests where aircraft are involved such as. 



system compatibility and integration tests (ALGM, launcher, aircraft, 
avionics) , captive flight performance and reliability tests, missile 
launch and free flight tests. The aforementioned tests are commonly 
performed in the DT&E phase by the contractor (though the contractor 
rarely operates the aircraft) and the TECHEVAL activity, and during 
OT&E by OPTEVFOR. If tests are combined and replications avoided, 

T&E cost growth and schedule problems present with the Separate Test 
Programs in Series should be significantly reduced. 

2. Applicable T&E Management Problems 

In addition to those problems alluded to in the preceding para- 
graphs, a number of significant problems remain, primarily in the 
areas of planning and resource and schedule constraints. Management 
of diverse activities and functions and proliferation of requirements, 
as previously discussed, will continue to be a problem but do not 
require further amplification here. The problem of redundant testing 
will be ameliorated to some extent by the combined testing approach, 
a. Planning 

The immediate problem in planning for combined testing 
stems from the OPNAVINST 3960.10 prohibition on combining TECHEVAL and 
OPEVAL, since these test programs appear to be quite similar. This 
prohibition relates to the basic purpose of DT&E and OT&E, i.e., DT&E 
is designed to assure that the contractual specifications have been 
met and OT&E determines whether the system fulfills the desired func- 
tions in an operational environment. OPTEVFOR also contends its evalua- 
tion is objective and unbiased, whereas TECHEVAL is not because PMTC 



is part of the Developing Agency (NAVAIR) . The Program Manager and 
other T&E planners must keep in mind that when combined testing is con- 
ducted , OPNAVINST 3960.10 requires that the necessary test conditions 
and test data required by both the DoD component developing agency and 
the OT&E agency must be realized. This may be difficult to accomplish 
because DT&E tests are usually structured to hold many parameters con- 
stant, isolate others, and allow measurement of specific quantities 
of interest. COMOPTEVFOR says this is usually not possible in OT&E and 
it is often not even possible to specify what OPTEVFOR wants to measure. 
The objective is often to create conditions as close to combat condi- 
tions as possible and watch what happens [Ref. 13] . 

A significant problem to the T&E planners (with DT&E and 
OT&E combined) will be resolution of the criteria for determining whether 
tests have been successful or not, i.e., agreement on test objectives 
and technical measures of effectiveness will be difficult to achieve 
and whether or not to proceed to the next test or series of tests may 
be quite controversial. The Commission on Government Procurement [Ref. 
11] recognized this situation could exist and addressed the problem of 
applicability of engineering specifications to the operational environ- 
ment by recommending some form of early OT&E to make sure that engineer- 
ing specifications have operational value. 

Finally, OPNAVINST 3960.10 requires that the OT&E agency 
M insure that the combined test is so planned and executed as to provide 
the necessary operational test information; participate actively in the 
test; and provide separate evaluation of the resultant operational test 
information' 1 . The planning and execution may be the most difficult 
aspect to assure. OPTEVFOR indicated that Program Managers seldom in- 
volve them early in the program, particularly, in the early planning, 
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but said the situation is improving. The major problem would appear to 
be the planning for the execution of the test. OPTEVFOR stresses an 
uncontrolled operational type of environment with operational personnel, 
i.e., people not specially trained, not test pilots, not technicians or 
engineers, but like typical officers and men in the fleet. 

b. Resource and Schedule Constraints 

T&E resources for combined testing are of special concern. 
Test facilities and ranges that would be required for ALGM combined 
testing are primarily R&D oriented. They possess good-to-excellent in- 
strumentation with a high degree of precision in measurement, but have 
very little flexibility with regard to the environment or method of 
application, i.e., not flexible in terms of application of operational 
tactics. The Blue Ribbon Panel noted problems in 1970 [Ref. 3], many 
of which are still with us today. Most of the facilities and ranges 
with state-of-the-art capability (i.e., maneuvering room and/or target 
complexes) are in great demand from a scheduling standpoint just to 
satisfy development (R&D) requirements. 

3 . Other Problems 

Several facets of the development process and DT&E make a com- 
bined DT&E and OT&E evaluation very unlikely. The very nature of the 
evolution of the ALGM product through DT&E (building block testing 
from components to systems) , limited design disclosure documentation 
to support it, and the evolutionary philosophy itself (design- test-re- 
design-retest) are inhibiting factors. Finally, OPTEVFOR finds that 
test objectives for development hardware tests are usually very limited 
and contractors and Program Managers are usually unwilling to test 
development hardware under operational conditions until late in the 
program. 
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4. Good Features 



The obvious good feature of the combined testing approach is 
the potential opportunity to conserve the T&E schedule and optimize the 
utilization of ALGM and test facility assets, thus improving the pro- 
gram's cost posture. Another facet is the requirement for an early 
positive involvement of the OT&E forces to assure compatibility of the 
ALGM engineering specifications with the operational environment in 
which the weapon will have to perform. 

Finally, the Defense Science Board observed in its 1977 report 
[Ref. 15] that a means should be sought to promote interaction, par- 
ticularly feedback from the OT&E to the developer. Combined testing 
would provide a very strong vehicle for this type of interaction. The 
Defense Science Board concluded "interaction among development test and 
evaluation and close contact with the user pays very important dividends 
in terms of money, time, and operational suitability". 

C. INTEGRATED TEST PROGRAM 

Another alternative available to the Program Manager for the ALGM 
TOE is the Integrated Test Program approach. The Integrated Test Pro- 
gram approach features performance of that amount of testing necessary 
to confirm compliance with specification requirements and verify opera- 
tional effectiveness and suitability. Close coordination, interface 
and pooling of information/data by all T&E participants is necessary 
to cumulatively build the net data base with each successive program 
phase. Properly executed, the Integrated Test Program approach mini- 
mizes ALGM and other asset requirements. The Integrated Test Program 
differs from Combined Testing in that the DT&E and OT&E testing is not 
conducted together or simultaneously but is conducted in a fashion 
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similar to the Separate Test Programs in Series, exclusive of the repeti- 
tion and redundancy and with an operational flavor provided by early, 
continuing OT&E participation. 

1. Management Approach 

The intent of the Integrated Test Programs approach is to work 
within the requirements and guidelines of OPNAVINST 3960.10 while 
vigorously controlling costs and schedule and because of this, much of 
the Management Approach information relative to the Separate Test Pro- 
grams in Series in applicable. The descriptions of DT&E , OT&E and 
their respective phases are still apropos and utilization of contractors, 
participating field activities, and other support activities is still 
required. 

The early stages of the program, prior to Milestone I, are 
critical to the success of the Integrated Test Program. It is impera- 
tive that the Program Manager or his designee for T&E , enlist the 
support and active participation of all T&E participants in the prepara- 
tion of ALGM specifications appropriately couched in operational termi- 
nology, followed by preparation of the TEMP for CNO approval just prior 
to Milestone I. The Program Manager must make it clear to all T&E 
participants that repetition and redundancy are not usually conducive 
to a timely and cost effective T&E program. He must emphasize that 
testing is required to have an operational flavor beginning in the 
early stages and will be building block in form, with data to be accumu- 
lated through successive stages of the program. Having laid this 
groundwork, the three basic test programs of the Integrated Test Program 
appear as follows: 
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a. Contractor Program 



DT-I through DT-III as described in section III C 1, i.e., 
the building block testing approach from major component to ALGM section 
and on to ALGM level culminating with Contractor Demonstration Tests, 
inc luding: 

(1) Part and component qualification, assembly, and sub- 
system tests (accelerated life, overstress, and design 
evaluation and qualification) , sub-system and section 
integration and compatibility tests. 

(2) ALGM system level performance tests on the ground 
(chamber tests), on board the aircraft, and in the 
captive carry mode concluding with several missile 
launches from aircraft, 

(3) reliability demonstration for compatibility with speci- 
fication requirements (ground and airborne testing for 
captive carry MTBF, launch and free flight to target 
reliability) at the required confidence levels, 

(4) maintainability demonstration for compatibility with 
specification requirements, 

(5) environmental and safety qualification (including 
electromagnetic compatibility) to verify compliance 
with specification design requirements. 

OPTEVFOR and TECHEVAL personnel should support the Program Manager and 
contractor in establishing specification requirements responsive to 
the threat with user inputs which are representative of the operational 
environment. After early verification of adequacy, continuing effort 
must be expended to maintain compatibility with the operational 
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environment, OPTEVFOR and TECHEVAL should support the Program Manager 
and his participating field activity Technical Manager in reviewing 
for government approval all Contractor Demonstration Test (CDT) plans, 
test procedures, and test reports and should actively participate in 
monitoring contractor CDTs to assure technical compliance and opera- 
tional adequacy . 

b. TECHEVAL Program 

Performed during the final sub-phase of DT-III and includes 
ALGM system level performance tests both on the ground (chamber tests) , 
on board the aircraft, and in the captive carry mode concluding with 
several launches, reliability data acquisition and evaluation as. part 
of the ALGM ground and airborne testing, sufficient to confirm ALGM 
adequacy and readiness for OPEVAL but not a repeat of the Contractor 
Demonstration Test. The intent is to build on the Contractor Demons tra 
tion Test reliability data base, ALGM configuration permitting. Main- 
tainability and supportability should be evaluated in concert with the 
testing described above. It is not necessary to repeat the Contractor* 
maintainability demonstration at this stage. Implementation of the 
foregoing TECHEVAL results in significant ALGM program cost savings 
over other T&E alternatives through conservation of T&E resources and 
schedule imp r ov erne nt . 

c. OT&E Program 

OT-II and OT-III, as described in section II C 2, is per- 
formed culminating with OPEVAL, which includes: ALGM system level per- 

formance tests on board the aircraft, on the ground and in the captive 
carry mode concluding with several missile launches, airborne data 
acquisition and evaluation of captive carry MTBF, and launch and free 
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flight reliability data as part of the ALGM ground and airborne testing. 
This should be sufficient to confirm operational effectiveness and suit- 
ability and when coupled with the accumulated applicable Contractor 
Demonstration Test and TECHEVAL data, should complete the development 
characterization of the ALGM and confirm its compliance with specifica- 
tion requirements and readiness for production. Suitable confidence 
limits should be employed and appropriate truncation techniques applied 
as necessary under the guidance of competent statistical support help 
to maintain statistical rigor. 

Maintainability, supportability and other logistics related 
ALGM system characteristics as well as compatibility, operability, human 
factors, and training adequacy should be evaluated. Tactics development 
and refinement should continue as necessary, 
d. FOT&E Program 

OT-IV is performed as described in section IIIc.3 and under 
the Separate Test Programs in Series approach. 

2 . Applicable T&E Management Problems 

Many of the problems facing the Program Manager are similar to 
those to be encountered in the prosecution of any T&E program. However, 
the nature of the Integrated Test Program and demands it makes on the 
Program Manager do create some unique situations. The spectrum of prob- 
lems identified in section IV have some degree of applicability but 
redundant testing with duplication of facilities, personnel, and their 
related costs is clearly reduced in significance, 
a. Planning 

The Program Manager will have to exercise strong powers of 
oersuasion to get the program T&E participants together with users eaily 
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and obtain the cooperation needed for planning vital to the structuring 
of the program budget and development of the baseline TEMP. This early 
cooperative effort will have to be sustained throughout the program to 
provide the impetus for implementation of the T&E program and for main- 
tenance and updating of the TEMP and the T&E portions of the DCP. The 
Program Manager will have to plan the acquisition of the ALGM design 
disclosure documentation package carefully and provide for configuration 
management in a timely fashion. The ALGM design disclosure documenta- 
tion package evolves with the design and is necessary to define and con- 
trol hardware configurations , and changes thereto, while undergoing T&E. 
This will support and lend credibility to the test programs by assuring 
knowledge of what was tested and will ease the evaluation of the effec- 
tiveness of changes. Also, if hardware configurations are known and con- 
trolled , development of a credible test data base will be made easier. 
This will be a difficult problem to overcome because, in the past, accord 
ing to NAVAIR, contractor, and participating field activity personnel, 
acquisition of the ALGM design disclosure data package has often been 
mishandled and not fully supported by Navy program sponsors and the 
Developing Agency. 

b. Proliferation of Requirements 

The Program Manager will still have to contend with problems 
due to requirements unique to certain participating field activities, 
test facilities, ranges and other agencies as indicated in section V A 2b 
A strong early planning effort with full participation and cooperation 
from the T&E community providing clear delineation of the program to be 
conducted and participant’s responsibilities with related budgets, should 
aid in the control process. The Program Manager requires the assistance 
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of higher Navy management and OSD in limiting the number of "special 11 
management reviews of the T&E program as well as other aspects of the 
program. This should be identified as an issue early in the program 
and brought to higher management attention during early program reviews 
and when the T&E program is presented for approval at Milestone I. The 
N'lARC report recognized this as a significant problem for the Program 
Manager, stating that, "his program is reviewed at various levels up 
through DoD by offices with little or no accountability. Hence, he can 
be "second guessed" by all concerned, yet he is the specific individual 
who must assign his assets in the most effective manner to execute the 
overall program" [Ref. 16]. 

c. Resource and Schedule Constraints 

Scarcity of ALGM assets and test facilities, possibly impact- 
ing T&E schedules, should be no more severe than under the other T&E 
approaches and may actually be alleviated by the Integrated Test Program. 
Fewer total ALGM assets and hopefully, fewer facilities are involved but 
specific targets and/or target complexes will be in demand and range 
instrumentation taxed. Maintenance problems may result because of higher 
utilization rates. 

Limited test flexibility and ability to conduct operational 
type testing on what primarily are still R&D oriented ranges will in- 
hibit the user-evaluator's IOT&E effort. Bowes [Ref. 9] also found 
that a major portion of the physical resources utilized by OPTEVFOR for 
its operational evaluation are controlled by the Developing Agency. The 
Integrated Test Program approach provides the mechanism for the develop- 
ment of plans for the utilization of these limited resources by OPTEVFOR 
and the DT&E agencies. Lack of control of the major portion of the 
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physical resources required for OT&E (instrumentation, ranges, data 
processing centers, simulators, and test aircraft) appears to reduce 
the independence of OPTEVFOR. 

d. Redundant Testing 

As previously states in section VC, redundant testing will 
be virtually eliminated as a significant problem in the ALGM T&E pro- 
cess by implementation of the Integrated Test Approach. Performance, 
reliability and maintainability testing will not consist of repeated 
lengthy and costly demonstrations of compliance with design require- 
ments but will be cumulative in nature, under conditions reflecting 
operational conditions agreed to in advance by all T&E participants. 
Credibility of the test base will have to be assured through proper test 
design and control, as well as early implementation of configuration 
management and a strong failure analysis and corrective action program. 
Implementation of the foregoing demonstrates a well thought out, 
plausible T&E approach, with reasonable test objectives and appropriate 
numbers of assets of known configuration with other necessary resources 
to progressively attain T&E objectives. 

e. Management of Diverse Activities and Functions 

The Program Manager will be severely tested as will the 
Navy's matrix organization in the implementation of the Integrated 
Test approach. The Program Manager must strive for early cooperation 
and acceptance by the participating NAVAIR design groups and test 
facilities, participating field activities, OPTEVFOR, and the contractor. 
The contractor may initially be balky when faced with the possibility 
of early exposure to ALGM test vehicles to at least a synthesized 
operational environment. Contractors and Navy Program Managers fear 
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that early , visible failures under anything approaching operational 
conditions, could result in program cancellation. Incorporation of 
operational requirements in design specifications will be difficult, 
requiring close coordination between all design groups and T&E program 
participants. The difficulty arises because of the limited success 
to date in translating operational requirements into terminology and 
units for inclusion in specifications for test and measurement [Ref. 20] . 

With regard to the TECHEVAL portion of the T&E program, the 
Program Manager may receive considerable opposition from within the 
NAVAIR T&E community and from PMTC because of the proposed reduced scope 
of TECHEVAL. However, when interviewed recently, the incoming AIR-06 
(formerly the PMA for ARI1 missiles, possessing major acquisition exper- 
ience) indicated that in his opinion, too much testing had been done in 
the past for testing's sake and that the T&E program has to be tailored 
to the overall weapons acquisition process. He also suggested that 
early planning was key and would enhance the program's stature with OSD. 

3 . Other Problems 

The Program Manager will have to assure early efforts on the 
part of the potential ALGM users, OPTEVFOR, NAVAIR and contractor design 
groups, in the development of test envelopes with respect to the mission 
profile from which T&E test plans will evolve. Mechanical model ALGMs 
will have to be used for aircraft captive carry, laboratory environmental 
determination efforts, and compatibility with designated and potential 
user aircraft. 

Another concern for the Program Manager will be the need to make 
DD (T&E) aware of the intended approach, prior to Milestone I if possible, 
and take the necessary steps to retain his understanding and concurrence 



85 



throughout the program. PMA-242 (the Anti-Radiation Missile Program 
Manager) , in an interview, indicated that periodic detailed briefings 
to DD (T&E) have served to promote an understanding and acceptance of 
lower level testing, simulations, and the fact that the totality of 
the worth of an ALGM system should not be based on free flight per- 
formances alone. DD ( T &E ) may also require assurance of OPTEVFOR's 
independence and their continued ability to be objective, even though 
involved very early in the program and to the extent required. 

4. Good Features 

Implementation of an Integrated Test Program affords an oppor- 
tunity to the ALGM Program Manager to alleviate the T&E impact on 
overall program costs and to exert better schedule control. The Defense 
Science Board indicated " there may be a potential to reduce the large 
demands of reliability and maintainability testing if the various 
agencies responsible for such testing could develop coordinated and in- 
tegrated test plans. That is, factory chamber testing, developmental 
testing under operational conditions, and operating testing all provide 
opportunity for collecting R&M data" [Ref. 15] . The Integrated Test 
Program approach should preserve the distinct DT&E and OT&E purposes 
and assure that appropriate personnel are doing the testing required 
under each type of testing. 

Greater involvement of OPTEVFOR should: 

a. help to alleviate problems with regard to development and 
review of incorporation of operational requirements into 
design specifications and translation of these requirements 
into tests, 

b. bring consistency in terms of test requirements, test 
methods, and test data acquisition and interpretation, 
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c • make it easier to stay abreast of the environment in which 
the threat will be encountered, and 
d. help make DT&E test results more credible and acceptable 
to DD(TSE) and the DSARC committee. 

The Integrated Test approach should also help to alleviate 
Defense Science Board concerns with regard to repetitive and redundant 
testing and NMARC concerns regarding an uneconomical amount of testing 
and testing beyond the capability of the hardware. And finally, the 
Integrated Test approach should provide an auditable trail of require- 
ments versus performance which has been of great concern to both 
DD (T&E) and the NMARC. 

0. SYSTEM LEVEL FUNCTIONAL TEST PROGRAM 

The last alternative or approach to be considered is that of the 
System Level Functional Test Program. Proponents of this method require 
that testing be accomplished at the ALGM system level on the assumption 
that the ALGM system and all of its parts can satisfactorily demonstrate 
required capabilities and be qualified as a unit. This approach is 
usually suggested when the item to be tested is not very complex, as 
a viable alternative in a crisis situation, or as a "get well" remedy 
when schedule slippage and cost growth is first detected in DT&E . The 
intent in the testing mcde is to perform as few tests as possible, con- 
serve ALGM test assets, have the shortest possible development T&E 
schedule, with minimum cost. Unfortunately, avionics equipments 
("black boxes”) are most often cited as representative of hardware 
qualified for service use in this fashion rather than existing ALGMs; 
also, most of the equipments cited were developed under quick response 
contracts during hostilities in southeast Asia and were few in number. 
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This approach is an extremely optimistic one. It would appear, on the 
surface, to reduce ALGM asset requirements, provide a shorter develop- 
ment schedule than any of the others considered, and therefore be cost 
effective . 

1. Management Approach 

The management approach is quite similar to that of the Separate 
Test Programs in Series or Combined Test Programs with the exceptions 
as follows: 

a. Contractor Program 

The contractor portion of the DT&E phase essentially discards 
the building block approach to hardware development, i.e., seeks to 
avoid the individual performance, reliability, and environmental quali- 
fication of critical parts, components, and assemblies. This approach 
also avoids or minimizes sub-system testing and tries to accomplish 
interface evaluation and compatibility verification at the time ALGM or 
system level testing is conducted. Performance, reliability, and en- 
vironmental qualification (i.e., Contractor Demonstration Tests) are 
performed at ALGM level, to ALGM system specifications. 

b. TECHEVAL Program 

The TECHEVAL is planned and conducted as with Separate Test 
Programs in Series (section VA 1 b) but without the prior testing and 
verification foundation usually acquired through the contractor program. 

c. OTSE Program 

OT&E is planned and conducted as previously indicated (i.e., 
Separate Test Programs in Series, section VA 1 c) . 
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2 . Applicable T&E Management Problems 



a. Planning 

Planning is extremely limited and optimistic to begin with 
(testing at ALGM level only and no contingency plans for failure) and 
problems which occur at ALGM system level are much more difficult to 
diagnose, analyze, and correct. Failures and related delays, redesigns 
and corrective action assessments with retests can consume significant 
amounts of time, impact schedule and costs severely, and cannot be 
planned for. 

The planning problem is even more basic than the previous 
paragraph would indicate. In today's acquisition environment, the 
Program Manager and his staff would probably encounter problems in 
arriving at an agreed upon TEMP approach with COMOPTEVFOR and doubtless 
would experience great difficulty in getting DD (T&E ) acceptance and/or 
concurrence to this approach unless there were unusual mitigating cir- 
cumstances, i.e., national emergency, significant technological gain 
that might be lost without a maximum concurrency type of approach, 
or some similar situation of urgency. 

b. Proliferation of Requirements 

Problems here are essentially the same as with Separate 
Test Programs in Series. 

c. Resource and Schedule Constraints 

Even with the limited asset approach indicated, certain 
major components and sections of the ALGM will require at least limited 
qualification and man-rating (proof of safe-for-launch from manned 
aircraft) prior to introduction into ALGM system level testing. One 
shot devices, batteries, fuzes, warheads and propulsion systems 



89 



(rocket motors) fall into this category. Until the limited qualifica- 
tion is accomplished, system level testing will be inhibited signifi- 
cantly. 

With the optimistic limited asset situation, there is little 
margin for error. There could be a severe resource problem with major 
schedule impact, as a result of failures, that could virtually bring 
the program to a standstill. Without adequate resources (ALGM assets) , 
problem evaluation, corrective action determination and re-test cannot 
proceed. A requirement to build additional hardware (i.e., re-start 
fabrication process) could effectively cause termination of the program. 
Finally, according to the Defense Science Board [Ref. 11] , hardware 
problems, a shortage of funds and schedule problems inhibits thorough 
risk assessment when problems occur. Without the risk assessment, the 
Defense Science Board says the DSARC will be faced with complex technical 
issues requiring a detailed review and may not permit progression to 
succeeding milestones. 

d. Redundant Testing 

These problems will essentially be the same as encountered 
with Separate Test Programs in Series (section V A 2 d) . 

e. Management of Diverse Activities and Functions 

Accent on minimum testing may create problems with certain 
activities and functions as discussed under V D 1 a; otherwise, problems 
will be similar to those of Separate Test Programs in Series. 

3 . Other Problems 

It is virtually impossible to stress all parts in an ALGM to 
system design limits much less get all parts to function as they would 
under mission conditions, while under system test conditions, due to 



90 



testing limitations posed by laboratories, facilities, and equipment. 
Therefore, part, component, assembly, sub-system tests leading to a num- 
ber of system tests is the optimum way to proceed. Even with the 
building block approach, failures occur, but with testing at ALGM level 
only there will be failures that will be extremely difficult to fault 
isolate and some may even go undetected until subsequent phases of 
testing where the true cause of failure may be masked. 

The 1974 Defense Science Board report [Ref. 14] stated, "Signi- 
ficant time and money will be saved if each component, each sub-system, 
and the full system are all tested as thoroughly as possible in the 
laboratory". The report went on to say, "Whenever field testing is ex- 
pensive compared with laboratory and simulation testing (as in missile 
or aircraft/missile flight testing) , such testing should be conducted 
primarily for verification of design parameters or design performance, 
rather than to see whether or not a particular component or sub-system 
will work" . 

4 . Good Features 

As seen in sections V D 2 and V D 3, this approach suffers from 
its departure, in the DT&E phases, from the proven hardware building 
block approach advocated by the Defense Science Board and has a high 
probability of failure. There are no known or recognizable features 
significant enough to overcome the inherent deficiencies in this testing 
approach. It does not appear to provide the basis for a sound develop- 
ment approach and it increases the acquisition risks greatly. 
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In this section, four of several possible alternative program 
management approaches to T&E programs for ALGMs have been presented. 
Each alternative was analyzed in terms of its management methodology, 
applicability of management problems identified in section IV to each 
alternative, and other problems deemed relevant and unique to that 
alternative. Favorable features were also identified as part of the 
analysis for each alternative. These analysis findings, in terms of 
positive and negative features, are summarized in Table I. Any one of 
the initial three alternatives would be acceptable given the proper 
circumstances and resources. 
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VI. PROPOSED TEST PROGRAM STRATEGY 



Bowes [Ref. 9] cited less than optimal utilization of the Navy's 
T&E resources and found that the prevalent belief that "more testing is 
better, so let's continue to do more and not worry so much about the 
duplication" was a simplistic and costly solution to the Navy's TSE 
problem. Changes have occurred in Navy T&E subsequently, but current 
OSD, CNO, and NAVMAT directives and instructions have not been imple- 
mented by directives and instructions from the Developing Agency most 
responsible for Navy ALGM development. In the absence of specific 
promulgated directives and guidelines and direct ALGM program partici- 
pation by AIR-06 personnel, Program Managers are still faced with the 
beliefs and perceptions cited by Bowes. 

Consideration of the alternative approaches to conduct of the T&E 
program for ALGMs in section V indicates that, on balance, the Integrated 
Test Program approach provides the best opportunity for the ALGM Program 
Manager to conduct an adequate, cost-effective, T&E program while work- 
ing within the requirements and guidelines of OPNAVINST 3960.10. 

Figure 7 presents the organizations, and their respective functions, 
which must come together early in the program to assist the Program 
Manager in developing and implementing the ALGM T&E program. 

The purpose of this section is to build upon the Management Approach 
portion of section V C and propose a "Test Program Strategy" which will 
aid ALGM Program Managers in the development and implementation of an 
Integrated Test Program. The proposed strategy will preserve the DT&E 



94 



CO 

-P 

CO 



CO 

-p 

CO 

0 




95 



Figure ?. ALGM T&E Organizational Interfaces 



and OT&E purposes while reducing repetitive and redundant testing, pro- 
vide an auditable trail of requirements versus performance, and help 
to bring consistency to ALGM testing in terms of test requirements, 
test methods, and test data acquisition and interpretation . The In- 
tegrated Test Program, to which the proposed strategy is applicable, 
is as depicted in section V C 1 and consists of the Contractor, TECHEVAL, 
OT&E, and FOT&E programs as outlined. 

A. ELMENTS OF THE PROPOSED STRATEGY 

The proposed strategy to be employed in an Integrated Test Program 
consists of consideration and implementation of a number of activities 
under the major ALGM elements of: 

1. Planning 

2. Types and Categories of Tests and Resource Requirements 

3. Responsibility for and Conduct of Tests 

4. Information Flow, Assessment and Reporting 

The necessary actions to implement the strategy are developed in subse- 
quent paragraphs. 

1. Planning 

The purpose of the Integrated Test Program approach, as it is 
with any test program, is to perform an optimum amount of testing (i.e., 
that amount necessary zo confirm compliance with specifications and 
operational effectiveness and suitability) and no more. Repetition, 
duplication, and redundancy are to be minimized. Hence, planning has 
been emphasized as a critical element in any T&E program (section III H) 
but as pointed out in section V C 2, it is especially critical to the 
Integrated Test Program approach because of the very close coordination 
and interface required of different, widely dispersed organizations and 
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because integration or pooling of test data is necessary to successfully 
demonstrate compliance with test objectives while utilizing minimum ALGM 
assets and other resources. 

Areas of consideration where actions should be taken to improve 
early and follow-on planning and also to increase the overall test and 
evaluation efficiency are as follows: 

a. Operational Requirements vs Design Specifications 

A very important facet of the early planning strategy requir- 
ing user participation or representation is the development of operational 
or functional requirements suitable for inclusion in design specifications 
to which contractor designed and built ALGM hardware will have to perform. 
These requirements become the basic criteria for the T&E program and for 
acceptance of T&E results. Tactical employment of the ALGM should be 
considered with ample resources brought to bear to develop a mission pro- 
file for inclusion in the specification. The mission profile should re- 
flect ALGM operating duty cycles based on tactical usage and should also 
consider the peacetime vs wartime scenario where expenditure rates effect 
the number of cycles (catapult launch, captive carry, and arrest landings) 
that ALGMs may be subjected to. The mission profile should also reflect 
consideration of the ALGM launch envelope, aircraft maneuvers, cockpit 
workload, and interface with other systems on the user aircraft. Aircraft, 
launcher, ALGM interfaces should be considered and user experience in 
these areas should be reflected in the design specifications. Finally, 
the design and operational or functional requirements must be translat- 
able into test requirements which are agreed upon by the T&E participants 
and can be employed in DT&E and OT&E phases of the program to confirm 
ALGM compatibility with design requirements and military worth and 



utility . 



b. Test Hardware Considerations 



The quantity of ALGM test assets required to support the 
T&E program and the amount of time provided for successful completion 
of all testing is critical to program success. Early planning must be 
performed to provide sufficient time and resources for operational 
tests, particularly the IOT&E phase, because the production and deploy- 
ment decision is based upon IOT&E results. Correspondingly, later in 
the program, the Program Manager should not lose sight of the continuing 
need for test articles. There have been circumstances where, for various 
reasons, the technology has had to be pushed rapidly; the higher risks 
involved have required more thorough testing and/or greater resources. 
These programs, usually had to have more assets and have been expanded 
to longer times. Typical hardware and facility requirements for an 
ALGM program are addressed in VI A 2. 

c. Test Facilities and Instrumentation Requirements 

Before Milestone I, the test facilities and instrumentation 
requirements to conduct tests should be identified along with a tenta- 
tive schedule of test activities. AIR-06 personnel must play a key role 
in this effort because of its relationship to the Navy's Major Range and 
Test Facility 3ase (MRTFB) . They must keep in mind that OPTEVFOR is 
totally dependent upon others for their test facilities and instrumenta- 
tion and that, historically, test ranges have inhibited operational test- 
ing because of their primarily R&D orientation and inherent lack of 
operational flexibility. The capabilities of the test ranges and the 
adequacy of the facilities and instrumentation should be verified; 
alternative accroaches considered (i.e., other ranges should be identi- 
fied) and the need for instrumentation improvements or changes be 
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identified early in the program. Range and instrumentation limitations 
may place constraints on ALGM flight testing and targets warrant special 
consideration; most ALGMs cannot be adequately tested without targets 
that realistically simulate the threat. Considerable lead time may be 
recurred to provide the required target support; therefore, target re- 
quirements to support the tests must be identified early, 
d. Test Configuration Control 

Proper ALGM and component configuration identification and 
control throughout the test program is essential to the creation and 
maintenance of a credible test data base. If it is not known what was 
tested, the test results are not too significant. Early planning will 
be required to develop the appropriate means of recording ALGM and/or 
component configuration (i.e., log books or similar methods) and for 
relating the configuration to test results. The recording means must 
be compatible with different working environments and users, i.e., 
laboratory and test range, contractors, participating field activities, 
and OPTEVFOR. Conditions and component configuration during development 
tests should be determined by the primary objectives of that test. 

Whenever a non-operational configuration (e.g., ALGM with 
inert warhead and/or rocket motor) is dictated by early test require- 
ments, tests should not be challenged by the fact that the configuration 
is not operational. Where tests are run with substitute parts, proce- 
dures should provide for recording the fact and ensuring that necessary 
retesting is done with the correct components. When testing is delayed 
because of the non-availability of critical sub-system components, off- 
the-shelf interim components may be used as substitutes until the proper 
components are available. As long as the off-the-shelf components can 
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function acceptably within a defined range of interest, the rest of the 
system can be tested, thereby facilitating the progress of the test pro- 
gram, On the other hand, demonstration and acceptance tests, as well 
as tests intended to evaluate performance under operational conditions, 
should always be conducted under conditions as close to those anticipated 
in operations as possible, including present ALGM configuration. 

The reliability data base is especially susceptible to prob- 
lems created by poor configuration control. To pool or group reliability 
data, as desired with the Integrated Test approach, the hardware tested 
must be from a homogeneous population. This determination of homogeneity 
through exercise of change control will require particular care once ALGM 
prototype and pilot production hardware becomes available for testing 
and demonstration in DT-III and OT-III. If change control is ineffective 
or major design changes are dictated, there may be no other alternative 
than a formal demonstration test program at a later point in the program 
but prior to the release to production decision (Milestone III) . 
e. Updating IOTSE Planning 

The early planning required to support initial DCPs and Mile- 
stone I will require frequent updating to keep pace with a dynamic pro- 
gram. The participating field activity Technical Manager and contractor 
will usually take the lead in maintaining the currency of the DT&E program 
but the Program Manager or T&E coordinator may have to provide the 
stimulus for the update of IOT&E planning during the early RSD phases. 

Few ALGM system programs have had adequate user participation with the 
desirable continuity of personnel to minimize the problems of transition 
from DT&E to OT&E to deployment/utilization. The early evaluations may 
be largely restricted to mock-up exercises and user-evaluator participa- 
tion in the RSD. One of the user-evaluator's major activities should be 
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to use the day— to— day exposure to the R&D system as on— the— job training 
in preparation for the IOT&E program. Good IOT&E planning should cover 
the allocation of manpower spaces or billets, assignment of personnel 
and their training, equipment provisioning (including at least pre- 
liminary technical manuals), ground support equipment, spare components, 
launchers, and missiles. IOT&E planning may also include plans and/or 
requirements for instrumentation, ground tests, and flight tests of 
important profiles and will also provide for data collection, analysis, 
and reporting. The latter assumes even greater significance when the 
Integrated Test Program approach is followed and the reliability data 
base is to be cumulative during crucial DT&E and IOT&E tests and evalua- 
tions . 

f. Computer Software 

As ALGM designs have become more sophisticated, utilization 
of mini-computers has become more commonplace and the problem of T&E of 
the central processor unit and the associated software has grown. The 
Program Manager or T&E coordinator must ensure that software products 
are tested appropriately during each phase. Even though the computer is 
a critical component of the ALGM, software has often been developed more 
as an add-on than as an integral part of the overall system. Software 
requirements need the same T&E consideration as hardware requirements 
in the early program phases. It is more difficult to determine the 
status of completion of various phases of the software program (as com- 
pared to hardware programs) , so it is important to explore how contractors 
develop and test software programs. No standard procedure seems to be 
available within DoD for orderly T&E of software items, yet the increased 
percentage of ALGM development cost introduced by software makes the 
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establishment of a suitable procedure a matter of utmost importance. 

Any new procedure should provide for orderly program definition and 
for continuous testing and monitoring of the software program develop- 
ment, to provide assurance that adequate, efficient, reliable operation 
will be possible. Annex A to the Defense Science Board Report provides 
specific guidelines for tracking the development of computer programs 
essential to the functioning of weapon systems [Ref. 14] . 
g. System Reliability 

Inability to meet specified reliability requirements and 
poor operational reliability has characterized many recently developed 
ALGIls. It should be emphasized that poor reliability is not only a 
function of component failures but also is influenced greatly by 
failures induced by poor hardware design, poor software design, operator 
errors, wear out of mechanical components, and failure to appreciate 
the severity of operational environmental conditions. The Program 
Manager or T&E coordinator must require that realistic operational 
as contrasted with design (inherent) reliability requirements be defined, 
in terms of completing a mission of specified duration under stated 
mission (environmental) conditions, and that testing adequate to demon- 
strate achievement of these requirements be accomplished successfully. 

It is necessary that the improvement of reliability be a 
planned activity during the development phases of the program, that it 
be monitored during these phases, and its achievement proven by testing 
prior to the major production decision. Interim goals must be estab- 
lished, with tests devised based on these goals, to allow tracking of 
reliability growth throughout the program. This progressive attainment 
of interim goals toward the required reliability for the ALGM*must be 
reviewed at critical points or milestones of the program, as follows: 
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(1) At the time the Program Manager requests initiation of 
engineering or full-scale development (Milestone II) , he 
should be prepared to show a realistic reliability 
growth plan with sufficient test time and resources to 
achieve the program reliability requirements from initial 
and interim goals to final requirement, 

(2) At the time the Program Manager requests initiation of 
limited production (Milestone II A) , he should be 
prepared to show: 

(a) By contractor demonstration test results , the sys- 
tem has achieved , at a reasonable confidence level, 
the expected percent of the reliability requirement 
for the ALGM, where both confidence level and per- 
cent achievement are appropriate to the program 

at this stage, i.e., tracking projected reliability 
growth rate, 

(b) There still remains in the development program 
sufficient ALGM system testing time to carry on 
reliability growth from the point achieved to the 
program reliability requirement. 

(3) At the time the Program Manager requests authorization 
for full-scale production (Milestone III) , he should be 
prepared to show: 

(a) By contractor demonstration and accumulated 

TECHEVAL and IOT&E test results, the ALGM system 
has achieved, at a reasonable confidence level, 
the program minimum acceptable reliability require- 
ment. 
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(b) A management plan, test plan, and funds to utilize 
the remaining test time in the development program 
for a vigorous program of reliability growth. 

(c) A plan for some of the earliest production missiles 
to be allocated to early fleet test programs for 
reliability and life tests. These ALGMs should be 
required to accumulate many more ramp exposure 
hours and captive flight hours than usual prior to 
launch. In this fashion, early fleet test missiles 
can provide confirmation of ALGM reliability or 
provide early indications of production problem 
areas and possibly preclude dangerous situations 

or stand-downs of important capabilities, 
h. Environmental Determination Effort 

To confirm the operational environmental requirements con- 
tained in ALGM specifications and the defined mission profile and to 
support the development laboratory tests and operational tests, thorough 
measurements should be made, as early as hardware permits, to determine 
the actual environment in which the ALGM components must live during the 
transportation, captive flight, launch and free flight phases of the 
ALGM life cycle. These environmental measurements should be conducted 
during advanced development preferably and no later than early engineer- 
ing development, because of the potential impact on the basic ALGM design 
and on the test programs. Funds, hardware, and necessary time should be 
provided for the measurement effort. In-flight environments, especially 
vibration, temperature, shock and stress imposed during the operational 
use of che ALGM, should be measured and documented under operational 
conditions using an instrumented ALGM. The captive flight measurements 
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should be utilized to structure ground environmental and reliability 
tests for demonstration of compatibility with captive flight require- 
ments under controlled laboratory conditions. This has been done 
successfully on several recent ALGM programs (e,g,, D-2 version of 
Standard ARM, AIM-9L Sidewinder, and the Air Force* s Maverick), 
i. Test Plan Coverage 

It is important that test plans prepared by the different 
T&E participants support the objectives of the Integrated Test Program 
by assuring compatibility and credibility of data bases so that results 
are integrable or capable of being pooled, configuration identification 
and control so that it is known what was tested, and by assuring testing 
is not repetitive or redundant and that the "right" people are doing 
the testing. The Program Manager or the responsible reviewing activity 
acting for the Program Manager, should further assure that sub-system 
and ALGM system test plans are compatible with pre-established mile- 
stones and goals for ready assessment of program progress at a later 
date. Every test plan should reflect the approved TEMP where appropriate 
and include as a minimum, clear statements regarding: 

(1) The overall purpose of the test 

(2) Critical issues with respect to operational requirements 

(3) The major test objectives 

(4) The schedule of test milestone events 

(5) The major resources required 

(a) Test environments, facilities, and instrumentation 

(b) Operational environment 

(6) The organizations which will conduct the test program 

(7) The analysis and evaluation approach 
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(8) Data acquisition and analysis responsibilities 

(9) Reporting of failures, failure analysis responsibili- 
ties and procedures regarding resumption of testing 

Planners should ensure that the schedule will accommodate problems and 
that clear, well-defined milestones for review and commitment to the 
next phase have been defined. 

The implementation of the Integrated Test Program approach 
changes the traditional role of the TECHEVAL agency by minimizing 
additional environmental testing as a result of previously performed 
contractor demonstration tests and limiting the reliability testing to 
that supporting the ALGM technical performance evaluation (i.e., ground 
testing on the aircraft, captive flight tests, launch and free flight) 
and contributing to the overall reliability data base. To further 
assure compatibility with the Integrated Test Program approach and the 
consistency and adequacy of the data base, the TECHEVAL test agency 
should be required to prepare and submit to the Program Manager, for 
approval, formal test plans and procedures followed by a formal test 
report at the conclusion of TECHEVAL. GPTEVFOR personnel should support 
the Program Manager and his participating field activity Technical 
Manager in reviewing the test plans, procedures, and report and should 
actively participate in monitoring the TECHEVAL itself to assure com- 
patibility with operational requirements and consistency of data base. 

2 . Types and Categories of Tests and Resource Requirements 
a. Types and Categories 

The types and categories of tests relevant to ALGM T&E are 
discussed in section III.C and basically reflect the DT&E/OT&E structure 
of OPNAVINST 3960.10. Figures 8 and 9 depict DT&E and OT&E, 
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Engineering development 1. a. Performance tests [Lab (bench), flight line, DT-I through 

and design support tests captive flight, launch and free flight], DT-III. 

b. Integration and compatibili ty tests. DT-I, DT-III. 

c. Environmental tests [Environ, survey DT-I through 

(mechanical models), environ, shakedowns, DT-III. 
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Figure 8. ALGM DT&E Tests 
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respectively, for the Integrated Test Program approach. Sub-categories 
of tests are identified and the suggested phase for performance of the 
test is indicated. This approach as depicted, is in consonance with the 
Defense Science Board recommended hardware building block approach and 
limits redundant testing between the contractor, TECHEVAL agency, and 
OPTEVFOR. The common thread, in terms of testing, between the partici- 
pants is obtained through tests performed on board the aircraft, on the 
flight line, during captive flight, and the launch and free flight to 
the target to determine that the test vehicle meets its basic technical 
requirements. The strategy, when implementing the Integrated Test 
Program approach, is to optimize the acquisition of not only performance 
data but also environmental, reliability, maintainability, operability, 
and human factors data by appropriate planning, design of the tests, 
and participation by user-evaluators as well as the developing agency. 
Under conditions, with appropriate configuration management in effect 
and where no major re-designs after prototype development ALGMs are 
fabricated, the contractor performance tests conducted under operational 
conditions should yield data that is integrable with TECHEVAL and OPEVAL 
data. The data, acquired under these conditions, is that also relevant 
to the ALGM acquisition decision process and most desired by DD (T&E) for 
its review and presentation to the DSARC, because it reflects progres- 
sive attainment of reliability goals and, ultimately, specified require- 
ments under operational conditions [Ref. 14] . It also provides infor- 

/ 

nation relative to ALGM maintainability and operability and can also 
yield much data useful for human factors analysis (e.g., poor handling 
characteristics, poor placement of controls and readouts) . 
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b. ALGM Models 



Figure 10 depicts the different configurations of ALGMs that 
will be required during the different phases of the development program. 
The tactical ALGMs (live warhead) will be primarily used during OPEVAL 
and will also be utilized to a limited extent during TECHEVAL and contrac- 
tor testing. This limitation is due primarily to the need for acquiring 
telemetered performance and environmental data with an instrumented test 
section throughout the DT&E period but also is due to the difficulty and 
expense of conducting warhead shots and the paucity of target assets. 
Guided Test Vehicles (GTVs) are used to evaluate full seeker and control 
section capabilities in captive and free flight with the aid of the 
instrumented test section. Particular emphasis is directed toward seeker/ 
control interface and launch and guidance characteristics to the target. 
Flight Test Vehicles (FTVs) are used to evaluate aerodynamic features 
and flight control and autopilot designs, and usually do not include a 
complete or functional ALGM seeker section. Static Test Vehicles (STVs) , 
featuring an inert warhead and rocket motor but with full capability 
seekers and control sections , are intended for extended periods of 
laboratory investigation and testing, i.e., environmental evaluations 
and tests and reliability growth testing. The Environmental Determina- 
tion Vehicle (EDV) is specially instrumented to acquire temperature, 
vibration, shock, and acoustic data during captive flight on representa- 
tive user aircraft. The same EDV will be used in the laboratory to 
simulate in-flight conditions and to develop environmental and relia- 
bility test criteria and should be available throughout the development 
program in the event of application to other user aircraft or major 
internal ALGM changes occur. Dummy ALGMs are used primarily for handl- 
ing tests, container tests, evaluating interfaces, and static displays. 
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Figure 10. Development Test ALGM Configurations 



c. Suggested Test Sequences/Flows 

Figures 11, 12, and 13 provide suggested test sequences/flows 
for the contractor tests, TECHEVAL, and OPTEVFOR, respectively, to maxi- 
mize the impact of the Integrated Test Program approach. A sequence/flow 
is not shown for DT-I because during this experimental period test se- 
quence/flow is normally unstructured, has very few test assets, and 
usually features the use of hybrid ALGMs . For example, a recent Anti- 
Radiation Missile (ARM) program used an experimental seeker and parts of 
Sparrow and SHRIKE missiles along with a SHRIKE rocket motor for its 
early captive flight and FTV investigations. 

Also shown within the individual activity blocks are typical 
ALGM asset quantities for that particular group of activities. Such 
asset quantities should normally provide adequate numbers for required 
tests yet retain a modicum of flexibility in the event of test failure 
or other setback. For example, 14 advanced development ALGMs are shown 
distributed as follows: 10 for captive flight, launch and free flight 

testing, 2 for environmental survey tests followed by failure mode test- 
ing, and 2 for environmental determination flights and laboratory con- 
firmation testing. 

The important features to be observed from the Integrated 
Test Program approach are: 

(1) Progressive attainment of goals and requirements by 
the appropriate agency utilizing optimum assets. 

(2) As shown in Figures 11, 12, and 13, TECHEVAL and 
OPTEVFOR personnel participate in the contractor test 
effort, and OPTEVFOR participates in the TECHEVAL test 
effort to provide the proper operational user-evaluator 
influence on test structuring and performance. 
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Figure 12. TECHEVAL Sequences/Flows - DT&E ALGM Test 
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( 3 ) 



The contractor does what he has been contractually obli- 
gated to do with performance of development tests and 
Contractor Demonstration Tests (CDTs) . 

(4) A controlled, consistently planned, executed and acquired 
data base is established with the CDTs and built upon 

by TECHEVAL and OPEVAL with an increasing operational 
flavor through TECHEVAL and OPEVAL (where only tactical 
ALGMs are used) . 

(5) The TECHEVAL role is modified and limited to confirming 
ALGM specification compliance with ground tests on the 
aircraft and captive flights, launches and free flights 
and also performs a supportability evaluation (no 
duplication of already accepted laboratory environmental 
tests or reliability tests) . 

(6) Failure reporting, analysis, and corrective action 
determination and verification criteria should be 
uniformly applied and implemented throughout the 
development program by the Program Management staff. 

d. Resource Requirements 

ALGMs cover a broad spectrum from Short Range Air- to- Air 
(SRAAM) to High Speed Anti-Radiation Missiles (HARM) with air-to-air and 
air-to-ground capabilities. Many different kinds of facilities, chambers 
and targets or target simulators are required to support these ALGM T&E 
efforts. Because of this wide dispersion of requirements, the resource 
requirement will first be presented in a general discussion framework 
and then requirements for a typical ALGM program will be shown. Generally, 
resource requirements for ALGMs exclusive of T&E personnel requirements 
encompass the following: 
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( 1 ) 



Contractor facilities and chambers for sub-systems and 
ALGM system level performance, structural, and a full 
range of environmental tests. Figure 14 is typical of 
the environments to which ALGMs must be subjected, 

(2) Contractor GSE/PGSE to support the testing along with 
necessary tools, benches, stands, and handling equipment, 

(3) Government facilities for assembly, checkout, and inte- 
gration testing of ALGMs prior to captive flight test- 
ing and launch. 

(4) Ranges and range instrumentation compatible with the 
ALGM and its mission and possessing sufficient flexi- 
bility to permit exploration of the performance enve- 
lopes in an operational environment. 

(5) Test aircraft including using or ALGM carrying aircraft 
representative of that with which the ALGM will be 
deployed, chase aircraft for safety and photographic 
purposes, as well as drones for use until the ALGM 
rocket motor is man-rated. 

(6) Instrumentation pod or an on-board installation to 
support the captive flight environmental determination 
effort. 

(7) Target simulators, aerial targets, and target compexes 
for air-to-ground ALGMs. 

The foregoing is an abbreviated, but representative, indication of re- 
source requirements for ALGM T&E. Tables II and III reflect the 
resource and facility requirements for a typical ALGM development T&E 
program which are beyond those normally available for fabrication, 
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N = Nonoperating. 
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2 = Test to he conducted with dummy missile(s). 

3 = Conducted on short round (Seeker, Control, Inert Warhead 

Sections only). 



Figure 1^. ALGM Environmental Test Matrix 
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TABLE II. Government Test Facilities Requirements 
(Adapted from Ref, 21) 



Bas e/Center 

Naval Weapons Center 
China Lake, California 



Facility 

Missile Integration 
Facility (Bldg 31420) 

Til Ground Station (Shrike 
Assembly Bldg) 

Hangar 3 (NAF) 

MARTS Facility (Bldg, 31420) 

A-7C Project A/C Data 
Reduction Facility 
(Bldg, 31420) 

Missile Assembly Bldg. 

(Area R Test Facility Area) 

Echo Range 

George Range 

NODAC Facility 

Range Safety 

Propulsion Firing Bay 
Warhead Ranges 

A-7E Integration Labora- 
tory (Hangar 3) 

SNORT Facility 

Beacon Shop 
T-Range 

Anechoic Chamber 
(Bldg. 31420) 

Environmental Test Facili- 
ties (Michelson Laboratory) 

Analog Simulation 
Laboratory 



Purpose 

Missile Assembly 
Test 

it 

Aircraft Integration 
Test 

Seeker/Avionics Tests 
A-7C Data Reduction 

Final Missile Inte- 
gration and Test 

Captive Flight Test 

Captive/Firing Tests 

Data Reduction 

Command Destruct 
System Checkout 

Motor Static Firings 

Missile/Avionics 

Integration 

Missile/Avionics 

Integration 

Radome Rain Erosion 
Tests 

Transponder Tests 
Hot Gas Tests 
Seeker Tests 

Environmental Testing 

Control Section Tests 
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TABLE II. Government Test Facilities Requirements (cont'd) 



Base/Center 

Edwards Air Force Base 
California 



Naval Air Test Center 
Patuxent River, Maryland 

Naval Weapons Laboratory 
Dahlgren, Virginia 

Naval Ordnance Station 
Indian Head, Maryland 



Naval Weapons Station 
Yorktown , Virginia 

Naval Ammunition Depot 
Crain, Indiana 



Facility 
Hangar Space 

Lab Space 

APR-38 Integ Laboratory 
TM Station 

Launcher Jettison Facility 
Carrier Qualification 
Facility 

Hero Laboratory 
Warhead Range 

Explosive Ordnance Dis- 
posal (EOD) Facility 
Propulsion Firing Bays 
Environmental Test 
Facility 

Warhead Loading Facility 

Environmental Facilities 
Test Arenas 



Purpose 

Missile/Aircraft 

Integ 

Missile Assy Test 

Missile/APR-38 Integ 

Missile Tests 

Jettison Tests 
Carrier Launch and 
Recovery 

Hero Tests 

Warhead Design Proof Tests 
EOD Techniques 

Motor Qualification 

Environmental 

Tests 

Warhead Loading 
Techniques 

Fuze Qualification 



TABLE III. Government Test Aircraft and Target 
Requirements (adapted from Ref. 21) 



Platform 

TA-4J Aircraft (3/N 152848) 
A-7C Aircraft (B/N 156739) 
Chase Aircraft 
QF-86 Drone 

F-4G/APR-38 Aircraft 

A-7E Aircraft (Two Required) 



Purpose 

Missile captive and firing tests 

Weapon System captive and firing tests 

Missile firing tests, photography 

Missile firing tests, FTV-103 (Non- 
man-rated rocket motor) 

Missile captive and firing tests 

Weapon System captive and firing tests 



NWC Simulator Assets for Missile Firings 

Expendable targets available at NWCs George Range to support ALGM 
firings include simulations of: 



Targets and Target Complexes 
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assembly , and test at any qualified ALGM contractor's plant. Additional 
information with regard to T&E resources, facilities, and funding of 
these activities are contained in section III F and G and Appendix C 
of this thesis. 

3 . Responsibility for and Conduct of Tests 

Sections III D and E and V C of this thesis address the subjects 
of responsibility for tests and conduct of tests, respectively, and pro- 
vide general information with regard to the roles of the various T&E 
participants in ALGM development programs. The purpose of this sub-sec- 
tion is to extend the general information mentioned previously to the 
implementation of the proposed planning strategy for an Integrated Test 
Program and to suggest the assignment of responsibility to test activi- 
ties and facilities to conduct the tests for ALGM and major component/ 
sections. Figures 15 and 16 depict DT&E and OT&E tests respectively, 
and the test category numbers shown reflect test categories from Figures 
3 and 9 . 

4 . Information Flow, Assessment, and Reporting 

Section III J of this thesis indicates the relative importance 
of Information Flow, and Review to Navy T&E programs 

structured for ALGMs. The Integrated Test Program approach of section 
V C and the Proposed Test Program Strategy of section VI do not change 
the historical reporting requirements, responsibilities, or channels of 
information discussed in those earlier sections. It does change the 
traditional ALGM T&E approach (e.g., re-orientation of user-evaluator 
participation and deletion of some TECHEVAL laboratory testing). Thus, 
the Program Manager and COMOPTEVFOR will still be the focal points for 
reporting to higher authority within the Navy and OSD regarding DT&E 
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Figure 16. ALGM OT&E Test Conduct and Responsibility 



and OT&E, respectively. OSD will provide program reports with T&E 
information to Congress, as required. However, the information result- 
ing from the implementation of the suggested strategy should give in- 
creased emphasis to the operational environment while still providing 
evidence of specification compliance. 

Employment of the suggested strategy (sections VI-A 1, VI-A 2, 
and VI-A 3) should help ensure that appropriate information is available 
for reporting program status to the DSARC at Milestone II (release to 
rull-scale development), Milestone II A (release to pilot production) , 
and Milestone III (release to full-scale production) . The Program 
Manager can point to the planned program and show progressive achieve- 
ments relative to the plan and OPTEVFOR's IOT&E evaluations, prior to 
OPEVAL, should have greater meaning due to the increased emphasis on 
operationally oriented T&E. The pooled or integrated data base will 
provide the means to greater visibility for all program participants 
and reviewers, of problems that have occurred, steps taken to rectify 
the problems, and the effectiveness of the corrective action. This 
increased user-evaluator participation throughout the T&E program, the 
plan for and adherence to the progressive attainment of ALGM perform- 
ance and reliability goals/requirements, and the increased problem 
identification and corrective action visibility should enhance the 
ALGM T&E program credibility in the eyes of DD(TSE) and other reviewers. 
The resulting T&E information should be especially important and use- 
ful to DD (T&E) when evaluating and making his assessment of the ALGM 
T&E program to the DSARC . 
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B. ALGM T&E ASSESSMENT 



1 • Impact of New Requirements and Guidelines 

Several T&E efforts associated with recently developed ALGMs 
were examined to assess the impact of the new requirements and guidelines. 
Unfortunately , most of the programs were already under development by 
contractors when the new requirements and guidelines were promulgated , 
and, as a result, many deficiencies which might otherwise have been 
avoided have occurred, necessitating redesign, rework, and retest efforts. 
All of the foregoing obviously add to the ALGM development cost and 
lengthen the schedule, 
a. AIM-7F 

For example, the AIM-7F failed OPEVAL I in 1972 which re- 
sulted in a major redesign encompassing 60% of the existing modules. 
Subsequent to the redesign, the T&E effort was not significantly re- 
structured from its predecessor (i.e., still performed to old T&E re- 
quirements) , and a building block approach to missile qualification was 
not followed. The mission profiles had been found to be inadequately 
defined and the specifications and required testing did not reflect the 
operational environment in which the AIM-7F was to perform. Inadequate 
planning for T&E and poor coordination among T&E participants resulted 
in severe laboratory environmental test failures and failure attributed 
to inappropriate environmental conditioning caused ALGM flight failures. 

OPEVAL II was held in 1974 and AIM-7F did not pass because 
captive flight MT3F , while improved, was still not adequate and design 
specifications for successful guidance were not achieved. In fact, many 
of the same problems which occurred in OPEVAL I were repeated in OPEVAL 
II indicating the intervening redesign and T&E effort preceding OPEVAL 
was ineffective. If the contractor DT&E and TECHEVAL effort had been 



125 



properly planned, coordinated , and implemented/ most of the defi- 
ciencies would probably have been corrected prior to OPEVAL* 



A follow-on version of the AIM-7F is now under development* 

As a result of the preceding deficiencies/ the Congressional Armed Ser- 
vices Committees have taken a very specific and direct interest in this 
ALGM's T &E, inserting the following language in the FY 1977 appropria- 
tions bill/ 

"The Navy and Air Force are advised to insure a viable test pro- 
gram for the monopulse missile that will clearly demonstrate the 
ability of this missile to perform in an operational combat environ- 
ment. The Director/ Test and Evaluation/ is to provide a report to 
the Committee on Armed Services at the conclusion of the advanced 
development phase that describes the test plan, the environment 
(electronic countermeasures, etc.), the test conditions, and the 
test results and evaluations" [Ref. 22] . 

b. AGM-78D 

The AGM-78D Standard ARM ALGM presents a somewhat better pic- 
ture than the preceding due to the implementation of an improved T&E pro- 
gram embodying many of the new T&E requirements. Prior to the "D" ver- 
sion, Standard ARM missiles had been subjected to a limited, poorly 
planned and executed T&E effort. ALGM specifications were inadequate 
(in fact were based on the shipboard Standard Missile) and requirements 
were nor stated in operational terms (i.e*, did not reflect either the 
physical or tactical environment)* Qualification of components, which 
was seldom performed, was to inappropriate specifications. The DT&E 
effort orovided poorly for design data feedback and was characterized 
by poor failure analysis and reporting. Contractor Demonstration Tests 
(CDTs) were performed for information only, and there was poor follow- 
up and little corrective action when failures occurred. There was little 
or no coordination between the T&E participants; consequently, when 
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TECHEVAL was performed/ ALGMs were overstressed and failed due to misin- 
terpretation of the missile specification and lack of a detailed test 
plan and procedure. Captive flight evaluations and ALGM launches ex- 
hibited execution problems and results were controversial. OPEVAL re- 
sults were similarly controversial and the missile failed to meet accept- 
able standards resulting in a recommendation to the CNO that the AGM-78B 
and C not be deployed. They were deployed, however, because of condi- 
tions in southeast Asia. 

The AGM-78D program, with strong leadership from the Navy 
Program Manager and his participating field activity Technical Manager, 
developed and implemented an improved T&E program starting in 1972 along 
with the redesign effort. A mission profile was jointly developed by 
the Navy and the contractor based on flights simulating tactical missions 
with partially instrumented ALGMs, and the ALGM specifications were 
modified accordingly. The building block approach to component, assembly, 
section, and ALGM qualification was employed culminating with a formal 
First Article Approval Demonstration Program where environmental and 
reliability demonstration under simulated mission conditions occurred. 

The Navy TECHEVAL agency and the Air Force Test and Evaluation Command 
to perform an OPEVAL type of test called AFEVAL (under a joint test 
agreement in which OPTEVFOR did not participate) were involved in the 
DT&E planning and monitored the Contractor Demonstration Tests. 

The TECHEVAL was planned and executed by the TECHEVAL agency 
without OPTEVFOR participation. Asset and funding limitations were 
present but the TECHEVAL agency performed some repetition of the Con- 
tractor Demonstration Tests (i.e., laboratory environmental and ground 
reliability tests) . Resource problems related to range instrumentation 



127 



and target availability were evident, but the results of the captive 
flight and launch performance and reliability evaluations confirmed 
compliance with the specifications. The results of the captive flight 
reliability evaluation and the contractor simulated captive flight 
reliability demonstration were remarkably similar. 

The AFEVAL, planned and conducted by the Air Force under 
the joint test agreement, exhibited results similar to TECHEVAL. Some 
problems related to ground handling and Peculiar Ground Support Equip- 
ment were encountered and operational testing flexibility was limited 
by the necessity to use Research and Development ranges and a shortage 
of some resources, but ALGM operational effectiveness and suitability 
were demonstrated. 

The foregoing is a classic example of the situation where 
we never have enough time and money to conduct a T&E program properly 
the first time but always seem to be able to find the time and money 
to do it a second, third, and even a fourth time. 

2. Post OPNAVINST 3960.10 

Currently, there is an ALGM development program underway that 
has the opportunity to develop and implement a T&E program in accordance 
with the new directives and guidelines of section III. The AGM-38 HARM 
(High-speed Anti-Radiation Missile) is currently in advanced development 
and has a system integration contractor under NAVAIR contract. A develop- 
ment specification has been prepared with performance and reliability 
requirements specified in terms of mission conditions. An extensive 
T&E planning effort has taken place with considerable effort on the part 
of the Navy Program Manager and the other T&E participants to develop a 
TEMP in accordance with the requirements of OPNAVINST 3960.10. 
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Examination of the HARM preliminary TEMP [Ref. 14] reveals that 
the planned T&E program is moving in the direction of the Integrated 
Test Program proposed in this thesis. Positive indications are extensive 
participation of all T&E participants in the planning of the T&E program 
and integration of IOT&E into the DT&E effort by having OPTEVFOR (VX-5) 
as the operator in a significant number of captive flights and ALGM 
firings. OPTEVFOR has monitored all advanced development firings and 
participated in all design reviews and Integrated Logistic Support' 
Planning conferences as has the TECHEVAL activity. A mission profile 
has been developed and is being confirmed by an environmental determina- 
tion effort consisting of captive flights with instrumented missiles 
followed by laboratory evaluation. The mission profile is being used 
to structure the simulated captive flight environment for the reliability 
growth test effort beginning in the engineering development phase and 
continuing through prototype development. A preliminary view of the 
growth test plan has been presented to NAVMAT and DDR&E (during a manage- 
ment review) and was found acceptable. 

During the prototype development phase of full-scale development, 
hardware and aircraft installations will be shared by DT&E and IOT&E. 
Contractor Demonstration Tests for specification compliance are planned 
for this period including laboratory demonstration of captive flight 
reliabili ty . 

The TECHEVAL will be conducted utilizing pilot production ALGMs • 
The current TECHEVAL plan includes ground chamber performance and 
reliability testing as well as captive flights to confirm specification 
compliance. There will also be an extensive supportability evaluation. 
The reliability testing (ground chamber and captive flight) will be to 
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demonstrate the required specification minimum acceptable MTBF at the 
90% confidence level. 

OPEVAL will include joint Navy /Air Force testing to examine 
operational effectiveness and suitability. OPTEVFOR will conduct a 
captive carry reliability evaluation and will pool OPEVAL data with 
previous DT&E/IOT&E data in this assessment. Approximately 1000 hours 
of ALGM captive flight operation is planned during the joint testing 
effort. 

In summary, the AGM-88 HARM ALGM T&E program approaches the 

proposed Integrated Test Program of this thesis. However, the HARM 

* 

program diverges from the Integrated Test Program by the TECHEVAL 
planned conduct of environmental tests and TECHEVAL will apparently 
not pool its data with contractor and other DT&E/IOT&E data. OPEVAL 
test data will be pooled, as previously mentioned, but a significant 
number of ALGM operational hours are to be acquired committing signi- 
ficant numbers of ALGM assets, range facilities, aircraft, and 
necessary support personnel for an extended period of time. 
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VI I . CONCLUSIONS AND RECOMMENDATIONS 



A. CONCLUSIONS 

consideration of the Navy’s ALGM T&E requirements, management 
problems, and alternative approaches to ALGM T&E management leads 
to the following major conclusions: 

!• successful outcome of the T&E program is largely dependent 

upon the managerial ability of the Program Manager to coordinate 
the diverse activities and resources all of which are not under 
his direct control. 

2. Excessive duplication exists in testing performed by major 
participants in the Navy ALGM T&E process. 

3. Operational test and evaluation agencies do not participate 
adequately in early T&E planning. 

4. Although still a relatively new requirement, preparation and 
implementation of TEMPs for some ALGMs and other air weapon 
systems are lagging behind program development schedules and 
OPNAVINST 3960.10 guidelines. 

5. A number of high level panels and commissions have performed 
studies of T&E over the past decade and have made similar rec- 
ommendations for improvement of the process and its management. 
While policy changes have been made, many of the management, 
resource and operational problems remain. 

Other significant conclusions are: 

1. Appropriate test design utilizing statistical design of exper- 
iment techniques is not being utilized by ALGM T&E participants 
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resulting, in some cases, in excessive and inefficient testing 
as well as improper interpretation and utilization of test data. 
Lack of definition of or delay in defining ALGM mission profiles 
leads to improper and often inadequate testing of ALGMs. 

3. Central management of NAVAIR T&E activities needs to be strength- 
ened to reduce costly duplication of facilities and capabilities 
among the Navy's Major Range and Test Facility Base for air 
weapons. 

4. Program Managers may be at a disadvantage when attempting to 
resolve, with COMOPTEVFOR, testing requirements reflected in 

the TEMP because of their relative positions with respect to CNO. 

5. OPTEVFOR is dependent upon the Program Manager for most of its 
IOT&E funding and resources; therefore, the structure of the 
OT&E effort may be adversely affected. 

6. OPTEVFOR needs to supplement its capabilities and resources to 
include analytical and technical expertise in order to meet its 
increasing testing responsibilities. This is especially true if 
the trend toward increased simulation and reduced expenditure of 
ALGM assets continues as a result of time and funding constraints. 

7. Software is given insufficient emphasis in early program phases, 
as no standard DoD procedure for T&E of software items exists. 

3. Unplanned reviews and requests for ALGM T&E data or additional 

tests by those concerned with program review and oversight affect 
program schedules and impact discretionally funds. 
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B. RECOMMENDATIONS 



1. T&E for new Navy ALGMs (ACAT I and II Programs) should be based 
on the Integrated Test Program approach as presented in this 
thesis . 

2. Early T&E planning as recommended in the "Proposed Test Program 
Strategy" should be emphasized on programs such as Short Range 
Air-to-Air Missile (SRAAM) and Medium Range Air-to-Air Missile 
(MRAAM) which are in advanced development. 

3. The Navy should pursue the idea of utilizing the skills of 
Navy Laboratories to support OPTEVFOR. The laboratories can 
provide the stability and continuity of personnel possessing 
appropriate technical expertise. 

4. NAVAIR should take steps to improve the timely communication, 
flowdown, and implementation of new T&E directives and guide- 
lines . 
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APPENDIX A 



Organizations with which Interviews were Conducted 

1. Office of the Secretary of Defense, Washington, D. C. 

DDR&E (T&E) 

2. Office of the Chief of Naval Operations, Washington, D. C. 

OP- 98 3 

3. Headquarters Naval Material Command, Washington, D. C. 

MAT-08 



4. Naval Air Systems Command, Washington, D. C. 

AIR-06 
AIR-620 
AIR- 242 

5. Operational Test and Evaluation Force 

Headquarters, Norfolk, Va. 

Air Development Squadron Five 

6. Naval Weapons Center, China Lake, Ca. 

Code - 06 
Code - 07 
Code - 35 

7. General Dynamics Corporation, Pomona Division, Pomona, Ca. 

Director of Defense Suppression 
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APPENDIX B 



Descriptions of DT&E, OT&E and PAT&E [Ref. 4] 

1- Development Test and Evaluation (DT&E) 

DTSE is primarily that test and evaluation planned, conducted, and 
monitored by the developing agency of the DoD component to demonstrate 
that the engineering design and development process is complete, that 
the design risks have been minimized, that the system will meet its 
perrormance specifications, and to estimate the system's military 
utility. DT&E includes testing of components, assemblies, sub-systems, 
software, hardware/sof tware integration, and advanced development and 
-ull-scale system level tests under various environmental conditions. 

Also tested is compatibility and interoperability with existing or 
planned equipments and systems. 

2 . Operational Test and Evaluation (QT&E) 

OT&E is that test and evaluation conducted to esimate the system's 
military utility, operational effectiveness (including survivability 
and vulnerability) , and operational suitability (including compatibility, 
availability, interoperability, reliability, maintainability, human 
factors, logistic supportability and training requirements) as well as 
the need for any modifications. In addition, OTOE provides information 
on organization, personnel requirements, doctrine, and tactics. It may 
also provide data to support or verify material in operating instruc- 
tions, publications , and handbooks. OT&E will be conducted in as 
realistic an operational environment as possible. OT&E is divided into 
two major parts: Initial OT&E (IOT&E) , which is defined as all OT&E 
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prior to the first major production decision; and follow-on OT&E (FOT&E ) , 
which is all OT&E after the first major production decision. 

3 . Production Acceptance Test and Evaluation (PAT&E ) 

PAT&E is test and evaluation of production items to demonstrate 
that the items procured fulfill the requirements and specifications of 
the procuring contract or agreement. It is the responsibility of each 
DoD component to accomplish the necessary PAT&E throughout the produc- 
tion phase of the system life cycle. 
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APPENDIX C 



Facilities for T&E of Navy ALGM [Ref. 5] 

1. Naval Ordnance Missile Test Facility, White Sands, New Mexico 

Its mission is to support the Navy guided missile and rocket programs 
including ground and flight testing and to participate in the operation 
of the Do D integrated missile test range at White Sands, a major national 
range. This activity reports to CDR, NAVSEA. 

2. Naval Air Test Center, Patuxent River, Maryland 

Its mission is to coordinate and perform T&E of aircraft weapons 
systems, their components and related equipment, conduct test pilot 
training, and provide technical advice and assistance to NAVAIR, con- 
tractors, etc. This activity reports to CDR, NAVAIR. 

3. Pacific Missile Range, Point Mugu, California 

Its mission is to provide range support for DoD and other designated 
government agencies for launching, tracking and collecting data in 
guided missile, satellite and space vehicle research development evalu- 
ation and training program and actual operations. This activity is a 
national range and reports to CDR, NAVAIR. 

4. Naval Weapons Center Ranges, China Lake, California 

Its mission is to provide ground ballistics, aircraft, explosive 
testing and supersonic track ranges for testing (R&D) the performance 
of fuzes, bombs, free-fall weapons, rockets, guided missiles, and other 
ordnance under actual operational conditions. This activity is part 
of a Navy in-house laboratory. 
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5 . Naval Missile Center, Point Mugu, California 

Its mission is to perform test, evaluation, development support and 
exercise engineering cognizance as assigned of naval weapons, weapon 
systems and related devices. This activity reports to CDR, NAVAIR. 

6. Naval Weapons Laboratory Ranges, Dahlgren, Virginia 

Its mission deals primarily with testing of Navy guns and mounts, 
although testing of some missile components also occurs. The range 
testing is subdivided into two broad categories; proof and acceptance 
testing, the object of which is to assure the quality, performance, 
safety and reliability of ordnance for the fleet; and developmental test- 
ing, the objective of which is to provide an experimental basis for new 
and improved weapons and systems. This activity is part of a Navy in- 
house laboratory . 

7. Naval Explosive Ordnance Disposal Facility, Indian Head, Maryland 
Its mission is to conduct RDT&E in technical matters for explosive 

ordnance disposal and render safe procedures for conventional and special 
weapons, guided missiles, biological and chemical munitions, equipment, 
both U.S . and foreign. This activity reports to CDR, NAVSEA. 
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